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Preface 
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and  infrastructure. 
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Executive  Summary 


The  National  Information  Assurance  Partnership  (NIAP)  is  a  joint  effort  of  the  National 
Institute  of  Standards  and  Technology  (NIST)  and  the  National  Security  Agency  (NSA)  to 
develop  and  promote  technically  sound  requirements  and  methods  for  evaluating,  infor¬ 
mation  technology  products  and  system  security.  The  long-term  goal  of  the  NIAP  is  to 
help  increase  the  level  of  trust  consumers  have  in  their  information  systems  and  networks 
through  the  use  of  cost-effective  security  testing,  evaluation,  and  validation  programs.  In 
meeting  this  goal,  the  NIAP  seeks  to: 

•  Promote  the  development  and  use  of  evaluated  IT  products  and  systems; 

•  Champion  the  development  and  use  of  national  and  international  standards 
for  IT  security; 

•  Foster  research  and  development  in  IT  security  requirements  definition, 
test  methods,  tools,  techniques,  and  assurance  metrics; 

•  Support  a  framework  for  international  recognition  and  acceptance  of  IT 
security  testing  and  evaluation  results;  and 

•  Facilitate  the  development  and  growth  of  a  commercial  security  testing  in¬ 
dustry  within  the  U.S. 

The  President’s  National  Strategy  to  Secure  Cyberspace  required  the  Federal  Government 
to  conduct  a  comprehensive  review  of  the  NIAP  to  determine  the  extent  to  which  it  is 
adequately  addressing  the  continuing  problem  of  security  flaws  in  commercial  software 
products.  The  DoD  and  the  DHS  tasked  the  Institute  for  Defense  Analyses  (IDA)  on  be¬ 
half  of  the  Federal  government  to  consider  results  of  current  policy  and  practices,  the 
general  efficacy  and  adequacy  of  current  capabilities,  and  the  subsequent  affordability 
and  viability  of  expanding  the  NIAP  program  to  all  federal  agencies.  The  direction  was 
to: 


•  Characterize  the  NIAP  intent  and  future  expectations,  conduct  fact  finding, 
and  develop  issues; 

•  Assess  impacts  of  selected  issues  and  generate  alternatives  and  options  to 
address  these  issues; 

•  Analyze  selected  issues  and  options;  and 

•  Recommend  option(s)  and  an  implementation  roadmap. 

Scope  and  Approach  of  Review 

To  ensure  coverage  of  all  relevant  issues,  the  scope  of  this  review  included  both  NIAP  as 
it  is  currently  instituted  and  the  broader  context  of  cybersecurity  within  which  NIAP  op¬ 
erates. 
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IDA  approached  the  NIAP  review  from  three  basic  analysis  viewpoints.  Since  Federal 
and  agency  policies  ultimately  dictate  requirements,  one  team  explored  policies  to  deter¬ 
mine  what  the  NIAP  has  been  directed  to  be.  A  second  team  examined  current  NIAP 
processes  in  order  to  observe  what  the  NIAP  currently  is.  A  third  team  delved  into  the 
expectations  of  various  NIAP  stakeholder  groups  to  learn  what  users  expect  and  need. 
This  was  done  through  interviews,  an  open  forum,  and  a  notice  in  the  Federal  Register 
soliciting  comments  on  the  NIAP.  The  results  were  then  combined  and  synthesized  by 
the  entire  project  team  to  reach  the  conclusions  herein. 

This  task  was  not  intended  to  establish  a  baseline  of  detailed  costs  and  specific  benefits  or 
to  detennine  the  statistical  significance  of  any  particular  measure  of  effectiveness.  Rather 
the  effort  was  a  review  of  the  current  posture  of  NIAP  relative  to  policies,  guidelines,  and 
standards;  relative  to  original  intent  of  the  partnership  and  current  practices;  and  relative 
to  a  broad  stakeholder  set  of  expectations  and  experiences. 

The  review  acquired  as  much  specific  data  on  actual  benefits  and  costs  as  possible;  how¬ 
ever  was  not  able  to  acquire  enough  evidence  for  a  rigorous  business  case  argument. 

Findings  Summary 

The  reviewers  organized  its  findings  along  the  dimensions  of  the  policies,  practices,  and 
expectations  used  to  obtain  information  and  to  assess  issues.  The  body  of  the  report  pre¬ 
serves  the  individual  details  of  these  findings.  A  synopsis  of  the  report  provides  an  inter¬ 
mediate  level  of  detail  following  this  executive  summary. 

This  review  affirmed  confusion  over  NIAP’s  role  and  scope  in  a  cybersecurity  context. 
Misunderstandings  spanned  five  basic  areas  of  characterization. 

•  Products  claiming  security  features,  functions  or  properties  (input  to  NIAP); 

•  Evaluation  processes  for  products  with  security  features,  functions  or  properties 
(NIAP  value-adding  activities); 

•  Implemented  systems  incorporating  evaluated  products  and  non-evaluated  prod¬ 
ucts  (use  of  NIAP  evaluations); 

•  Operational  outcomes  or  effectiveness  of  systems  with  security  expectations  from 
using  evaluated  products  (outcome/benefit  measurement  versus  NIAP  in¬ 
put/output  measurement);  and 

•  Equipment,  software,  and  expertise  to  perfonn  evaluations  and  their  limitations  in 
detennining  security  features,  functions,  or  properties  of  products  against  stan¬ 
dards  and  in  operational  use  (evaluation  tools,  techniques  and  infrastructure). 

The  overall  findings  in  each  area  are  summarized  as  follows. 

Policy  and  Policy-Related  Summary 

The  review  identified  and  examined  over  100  relevant  cybersecurity  related  policy  and 
guidance  documents.  Three  major  documents  provide  the  governing  policies  for  NIAP 
and  NIAP  specific  evaluations.  The  remaining  documents  establish  the  cybersecurity  en¬ 
vironment  in  which  NIAP  operates. 
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The  cybersecurity  policy  landscape  is  complex.  Although  product  developers  and  experts 
find  it  complex  they  are  better  equipped  to  know  and  judge  if  they  are  compliant,  capable 
of  conducting  product  security  evaluations,  and  the  utility  of  product  evaluations  in  the 
larger  environment  of  cybersecurity.  For  users  and  consumers  of  products  claiming  secu¬ 
rity  properties  and  promoting  security  features  it’s  a  much  more  difficult  endeavor  to 
know  and  appreciate  whether  a  NIAP  evaluation  is  valuable  or  meaningful  in  the  envi¬ 
ronments  or  configurations  in  which  they  are  incorporated  or  used. 

Practice  and  Practice-Related  Summary 

NIAP  activities  as  originally  scoped  remain  valid.  Little  was  discovered  against  the  gen¬ 
eral  proposition  of  testing  and  evaluating  products  against  established  standards  for  secu¬ 
rity  properties.  However,  NIAP  expenditure  priorities  versus  the  original  scope  of  activi¬ 
ties  (e.g.,  evaluations,  education  training  and  awareness,  and  research  and  tools)  have 
shifted  and  been  dominated  by  increasing  demands  for  evaluations  with  a  corresponding 
declining  budget  for  other  NIAP  activities. 

NIAP  as  designed  and  implemented  is  incomplete  to  fully  address  the  myriad  issues  and 
demands  of  cybersecurity  emerging  today.  However,  NIAP  is  accomplishing  a  major  por¬ 
tion  of  its  original  goals  with  limited  funding  but  funding  limitations  put  its  immediate 
future  in  jeopardy.  This  limited  funding  has  led  to  deficiencies  in  NIAP  relative  to  Protec¬ 
tion  Profiles,  strengthening  and  fixing  Common  Criteria  and  requirements,  development 
of  tools  for  evaluations,  and  adequate  linkage  with  Certification  and  Accreditation  (C&A) 
processes. 

Expectations  and  Expectations-Related  Summary 

Expectations  of  what  NIAP  can  accomplish  relative  to  the  totality  of  cybersecurity  de¬ 
mands  -  both  actual  and  emerging  needs  -  are  not  well-understood  and  often  based  in 
incorrect  perceptions  of  what  NIAP  is,  what  policies  govern  NIAP,  what  NIAP  does,  and 
varied  experiences  with  security  evaluations  in  contexts  that  may  or  may  not  engage  with 
NIAP.  With  an  environment  demanding  more  cybersecurity,  competing  ideas  arise  for 
what  NIAP  could  or  should  be  contributing.  No  reliable  evidence  was  found  that  alterna¬ 
tives  to  the  concept  of  NIAP  were  sufficiently  mature  to  be  competing  alternatives.  That 
is  to  say,  evidence  was  not  collected  that  some  response  other  than  NIAP  was  clearly  and 
measurably  better.  However,  many  ideas  on  ways  to  improve  the  results  and  practices  of 
NIAP  surfaced.  A  number  of  these  expectations  expressed  by  stakeholders  or  benefactors, 
lead  naturally  to  the  development  of  a  number  of  options  for  improving  NIAP  within  the 
cybersecurity  context.  These  options  spanned  all  expectations  expressed  by  the  commu¬ 
nity.  On  one  end  of  the  spectrum  of  opinion,  some  thought  there  was  no  need  for  NIAP 
and  it  should  be  eliminated.  The  other  extreme  expressed  a  need  to  move  to  a  “new  para¬ 
digm”  to  accomplish  cybersecurity  evaluations.  This  review  found  that  the  likely  reality 
for  NIAP,  when  all  comments  were  examined,  was  intermediate  to  these  endpoints. 

The  review  examined  over  750  individual  comments  to  identify  major  areas  of  issues  and 
specific  perceptions  and  expectations.  The  major  areas  of  issue  are  education  of  stake¬ 
holders,  research  to  obtain  adequate  tools  for  evaluation  and  alternate  fonns  of  assurance, 
applicability  to  critical  infrastructure  and  their  associated  information  systems,  questions 
of  product  evaluation/assurance  in  relationship  to  process  evaluation/assurance  and  com¬ 
position  of  products  (both  evaluated  and  unevaluated)  into  secure  systems. 
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Recommendations  Summary 

The  review  explored  6  options  in  response  to  the  findings  in  policy,  practice,  and  expecta¬ 
tions.  The  implications  of  these  options  were  developed  and  organized  as  follows. 

1.  Eliminate  the  NIAP  and  product  evaluations 

2.  Con  tinue  the  NIAP  in  its  current  form  (reduced  from  the  original  in  ten  t) 

3.  Restore  the  NIAP  to  the  original  intent  of  the  Letter  of  Partnership  between  NSA 
and  NIST 

4.  Modernize  the  approach  to  cybersecurity  evaluations  to  reflect  changes  in  the  en¬ 
vironment  since  its  creation  in  1998 

5.  Integrated  approach  to  cybersecurity  evaluations 

6.  Forwar-  looking  approach  to  cybersecurity  evaluations  (new paradigm) 

These  options  were  constructed  to  enable  each  (beginning  with  option  2)  to  build  on  the 
prior  option  in  tenns  of  implementation  increments.  Option  1  was  developed  as  a  re¬ 
sponse  to  expectation  that  NIAP  could  be  eliminated  without  replacement.  Option  6  was 
developed  as  a  response  to  expectation  that  a  totally  new  paradigm  was  needed  for  cyber¬ 
security  evaluations. 

The  report  recommends  the  selection  of  Option  3,  restoring  NIAP  to  its  original  intent. 
Option  2  is  required  to  achieve  Option  3,  but  option  2  only  addresses  the  increasing  de¬ 
mands  for  evaluations  without  regard  to  the  original  scope  of  the  partnership.  Option  3 
was  designed  as  a  response  that  restores  NIAP  to  its  original  scope  but  in  the  context  of 
cybersecurity  demands  of  2005.  Options  4  and  5  were  further  developed  to  expand  and 
extend  the  scope  of  NIAP  for  integration  with  existing  C&A  activities  of  Information  As¬ 
surance  (IA)  and  to  extend  this  integration  to  the  potential  (but  not  yet  proven)  needs  of 
cybersecurity. 

The  NSA  and  NIST  provided  specific  cost  estimates  to  implement  Options  2  and  3.  These 
estimates,  included  in  a  separable  Annex  to  this  report,  are  closely  matched  to  the  relative 
estimates  this  review  developed  for  implementing  the  options.  The  review  established  a 
baseline  by  using  an  existing  budget  and  the  quantity  of  evaluations  performed  for  the 
budget.  This  was  then  used  to  estimate  the  incremental  need  for  resources  assuming 
growth  projections  for  evaluation  demands  and  separate  estimators  were  used  to  develop 
resource  sizing  for  options  three  and  later.  The  synopsis  following  provides  intermediate 
details  of  the  recommendations  associated  with  choosing  option  2  and  3  to  implement 
going  forward. 

Conclusions 

The  review  of  NIAP  concludes: 

•  NIAP  is  providing  a  useful  and  significant  service 

•  The  purpose  of  NIAP  is  neither  widely  or  well-understood 

•  NIAP  is  complementary  and  supplements  Certification  and  Accreditation  (C&A) 
and  is  not  a  replacement  for  C&A 
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•  NIAP  needs  to  be  better  connected  to  and  integrated  with  C&A 

•  NIAP  should  provide  better  information  for  system  integrators,  system  engineers 
and  security  engineers 

•  NIAP  should  be  established  as  an  accountable  program  with  distinct  budget  lines 
and  supporting  research 

The  next  section  provides  a  synopsis  of  the  report  details  for  those  readers  seeking  an  in¬ 
termediate  description  of  the  review.  The  remainder  of  the  document  provides  the  details 
discovered  or  developed  during  the  review. 
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Report  Synopsis 


The  National  Information  Assurance  Partnership  (NIAP)  is  a  joint  effort  of  the  National 
Institute  of  Standards  and  Technology  (NIST)  and  the  National  Security  Agency  (NSA)  to 
develop  and  promote  technically  sound  requirements  for,  and  methods  for  evaluating,  in¬ 
formation  technology  product  and  system  security.  The  long-term  goal  of  the  NIAP  is  to 
help  increase  the  level  of  trust  consumers  have  in  their  information  systems  and  networks 
through  the  use  of  cost-effective  security  testing,  evaluation,  and  validation  programs.  In 
meeting  this  goal,  the  NIAP  seeks  to: 

•  Promote  the  development  and  use  of  evaluated  IT  products  and  systems; 

•  Champion  the  development  and  use  of  national  and  international  standards 
for  IT  security; 

•  Foster  research  and  development  in  IT  security  requirements  definition, 
test  methods,  tools,  techniques,  and  assurance  metrics; 

•  Support  a  framework  for  international  recognition  and  acceptance  of  IT 
security  testing  and  evaluation  results;  and 

•  Facilitate  the  development  and  growth  of  a  commercial  security  testing  in¬ 
dustry  within  the  U.S. 

The  National  Strategy  to  Secure  Cyberspace  required  the  Federal  Government  to  conduct 
a  comprehensive  review  of  the  NIAP  to  detennine  the  extent  to  which  it  is  adequately 
addressing  the  continuing  problem  of  security  flaws  in  commercial  software  products. 
The  DoD  and  the  DHS  tasked  the  Institute  for  Defense  Analyses  (IDA)  on  behalf  of  the 
Federal  government  to  consider  results  of  current  policy  and  practices,  the  general  effi¬ 
cacy  and  adequacy  of  current  capabilities,  and  the  subsequent  affordability  and  viability 
of  expanding  the  NIAP  program  to  all  federal  agencies.  The  direction  was  to: 

•  Characterize  the  NIAP  intent  and  future  expectations,  conduct  fact  finding, 
and  develop  issues; 

•  Assess  impacts  of  selected  issues  and  generate  alternatives  and  options  to 
address  these  issues; 

•  Analyze  selected  issues  and  options;  and 

•  Recommend  option(s)  and  an  implementation  roadmap. 

To  ensure  coverage  of  all  relevant  issues,  the  scope  of  this  review  included  both  NIAP  as 
it  is  currently  instituted  and  the  broader  context  of  cybersecurity  within  which  NIAP  op¬ 
erates. 

IDA  approached  the  NIAP  review  from  three  basic  analysis  viewpoints.  Since  Federal 
and  agency  policies  ultimately  dictate  requirements,  one  team  explored  policies  to  deter- 
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mine  what  the  NIAP  has  been  directed  to  be.  A  second  team  examined  current  NIAP 
processes  in  order  to  observe  what  the  NIAP  currently  is.  A  third  team  delved  into  the 
expectations  of  various  NIAP  stakeholder  groups  to  learn  what  users  expect  and  need. 
This  was  done  through  interviews,  an  open  forum,  and  a  notice  in  the  Federal  Register 
soliciting  comments  on  the  NIAP.  The  results  were  then  combined  and  synthesized  by 
the  entire  project  team  to  reach  the  conclusions  herein. 

This  task  was  not  a  rigorous  study  meant  to  establish  a  baseline  of  costs  and  specific 
benefits  or  to  detennine  the  statistical  significance  of  any  particular  measure  of  effective¬ 
ness.  Rather  the  effort  was  a  review  of  the  current  posture  of  NIAP  relative  to  policies, 
guidelines,  and  standards;  relative  to  original  intent  of  the  partnership  and  current  prac¬ 
tices;  and  relative  to  a  broad  stakeholder  set  of  expectations  and  experiences. 

The  review  attempted  to  acquire  as  much  specific  data  on  actual  benefits  and  costs  as 
possible;  however  was  not  able  to  acquire  enough  evidence  for  a  business  case  argument. 
This  should  not  be  perceived  as  negative  with  respect  to  the  value  stakeholders  perceive 
for  a  product  security  evaluation  capability;  but  rather  an  inherent  difficulty  of  obtaining 
the  necessary  data  and  its  correlation  with  effectiveness  of  resource  execution. 

Findings  Summary 

The  review  organized  its  findings  according  to  the  independent  reviews  of  the  policies, 
practices,  and  expectations  teams  used  to  obtain  information  and  to  assess  issues  of  each 
area.  The  body  of  the  report  provides  the  individual  details;  some  of  which  are  not  cov¬ 
ered  in  this  synopsis,  as  they  are  specific  to  a  sub-category  of  the  major  areas  explored. 

Policy  and  Policy-Related  (See  Chapter  3) 

Policies,  guidance,  standards,  plans,  and  directions  provided  the  basis  for  examining  the 
environment  of  NIAP,  NIAP  specific  roles  and  responsibilities,  and  the  basis  for  judging 
NIAP  performance.  After  examining  this  landscape,  the  policy  review  team  established 
and  organized  findings  into  5  themes.  These  themes  were  1)  cybersecurity,  2)  standards, 
3)  research,  4)  education,  training  and  awareness,  and  5)  acquisition.  These  permitted  an 
assessment  of  the  situation  relative  to  the  environment  (cybersecurity),  an  established 
baseline  of  requirements  (standards),  the  evolving  nature  of  tools  and  products  (research), 
the  knowledge  of  users,  developers,  and  other  stakeholders  (ET&A),  and  finally  the  con¬ 
straining  factors  that  enable  or  impede  the  effectiveness  of  product  evaluations  (acquisi¬ 
tion). 

Cybersecurity: 

•  The  complex  policy  landscape  and  lack  of  a  single  source  for  current  and  super¬ 
seded  policies  makes  it  difficult  for  Federal  Departments  and  Agencies  to  deter¬ 
mine  the  requirements  for  their  particular  situation. 

•  The  NIAP  was  created  by  agreement  between  two  agencies  in  different  Federal 
departments,  with  no  fonnal  recognition  by  OMB  or  Congress.  It  has  no  official 
standing  other  than  the  agreement,  which  can  be  modified,  rescinded,  or  termi¬ 
nated  unilaterally  by  either  of  the  two  parent  agencies. 

•  The  NIAP’s  budget  is  not  a  line  item  in  either  parent  agency’s  budget,  preventing 
detailed  oversight  of  the  budget  process  to  detennine  sufficiency  and  justification. 
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This  has  resulted  in  decreasing  available  resources  as  the  parent  agencies  address 
more  pressing  issues. 


Standards: 

•  The  National  Technology  Transfer  and  Advancement  Act  [NTTAA]  1995  re¬ 
quires  the  use  of  voluntary  consensus  standards  in  lieu  of  developing  Federal 
standards.  For  Federal  agencies  to  determine  which  standards  to  use,  requires  the 
mapping  of  existing  voluntary  consensus  standards,  Federal  standards  (FIPS)  and 
standards  from  the  different  communities  of  interest  to  determine  gaps,  conflicts 
and  overlaps.  To  this  date,  this  mapping  has  not  been  done,  or  if  it  has,  was  not 
evident  to  the  researchers  in  this  review. 

•  The  existence  of  the  communities  of  interest  results  in  differing  sets  of  potentially 
conflicting  and  non-interoperable  requirements.  While  Federal  Infonnation  Proc¬ 
essing  Standards  (FIPS)  issued  by  NIST  are  mandatory  for  the  Federal  govern¬ 
ment,  National  security  systems  (NSS)  are  exempt  from  them.  NSS  have  their 
own  standards  and  guidelines  as  does  the  DoD  and  Intelligence  Community.  An¬ 
nex  D  provides  more  detail  on  the  standards  for  these  communities  of  interest. 

Research: 

•  Current  levels  of  cybersecurity  research  funding  are  inadequate  and  fail  to  address 
current  cybersecurity  issues,  much  less  those  of  the  future  (such  as  grid  comput¬ 
ing,  distributed  intelligent  agent  systems,  distributed  knowledge  management, 
composable  systems,  and  systems  of  systems.) 

•  A  timeline,  schedule,  and  process  for  addressing  future  cybersecurity  concerns  are 
lacking. 

•  A  process  for  coordinating  government  efforts  and  allocating  research  resources 
for  cybersecurity  research  is  lacking. 

•  The  memorandum  establishing  the  NIAP  (see  Annex  G)  says  that  NIAP  seeks  to 
foster  research  and  development  in  security  tests,  methods,  and  metrics  but  to  date 
this  has  not  happened. 

•  While  some  portion  of  infonnation  security  research  advances  achieved  as  a  result 
of  the  Homeland  Security  Act  of  2002  may  be  of  use  for  the  NIAP,  no  process  is 
in  place  to  identify  those  results  or  to  transfer  them  into  the  CCEVS  process 

•  The  need  for  research  to  improve  the  quality  of  our  cyber  defense  and  defensive 
information  operations  is  widely  acknowledged  in  numerous  government  docu¬ 
ments;  however,  no  document  points  to  the  NIAP  as  a  tool  for  addressing  this 
need  or  directs  research  that  would  result  in  the  NIAP  being  able  to  address  this 
need.  In  the  few  cases  where  the  need  for  research  is  identified  in  a  document,  it 
is  usually  cited  in  relation  to  a  specific  threat,  such  as  mobile  malicious  code,  and 
never  addressed  toward  improving  the  capability  of  determining  if  software  is  se¬ 
cure  via  the  NIAP  process. 

•  While  the  Clinger-Cohen  Act  imposes  a  research  duty  upon  the  National  Science 
Foundation  (NSF),  there  is  no  mechanism  for  communicating  NIAP’s  needs  to  the 
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NSF  or  for  extracting  NSF  research  advances  and  employing  them  for  NIAP  pur¬ 
poses. 

Education,  Training  &  Awareness  (ET&A): 

•  The  training  documentation  produced  by  NIST  for  the  Federal  Government 
(SP800-16)  in  1998  was  issued  concurrently  with  the  establishment  of  the  NIAP. 
Although  a  model  program  when  it  was  issued,  the  changes  in  policy,  technology, 
management  of  Federal  IT  organizations,  and  organization,  require  a  similar  sig¬ 
nificant  update  to  this  document. 

•  The  level  of  detail  in  the  current  NIST  SP800-16  is  insufficient  to  ensure  an  ap¬ 
propriate  level  of  knowledge  for  either  the  NIAP  certificate  users  or  certifiers. 
This  detail  is  necessary  for  perfonnance  and  evaluation  of  performance  for  those 
functions. 

Acquisition: 

•  While  FISMA  documents  specify  the  security  controls  that  must  be  implemented 
in  Federal  unclassified  systems,  there  is  no  requirement  on  how  Federal  agencies 
must  choose  those  products.  The  requirement  for  acquisition  of  evaluated  IA/IA- 
related  products  only  exists  for  the  NSS  and  DoD.  Outside  of  this  no  acquisition 
policy  concerning  IA/IA-related  products  exists  for  Federal  departments/agencies 
since  the  Federal  Information  Resources  Management  Regulations  (FIRMR)  was 
rescinded  in  1996.  This  means  that  the  rest  of  the  Federal  Government  can 
choose  security  products  based  on  their  own  criteria  that  may  or  may  not  have 
been  evaluated. 

•  The  NIAP  process  leading  to  product  certification  does  not  directly  contribute  to 
systems  certification  required  by  statute  and  OMB. 

Practice  and  Practice-Related  (See  Chapter  4) 

The  practices  team  principally  found  that  NIAP  is  underperfonning  relative  to  original 
intent  due  to  two  driving  factors.  One  is  a  shortage  of  resources  to  accomplish  to  the  full¬ 
est  degree  all  that  was  intended  by  the  original  letter  of  partnership.  The  second  factor  is 
that  for  a  given  level  of  resources  applicable  for  the  scope  of  original  intent,  these  are 
rapidly  consumed  just  meeting  the  non-discretionary  requirements,  leaving  little  for  the 
remaining  scope  of  NIAP  intentions.  Simply,  other  priorities  are  crowded  out  by  the  in¬ 
creasing  quantities  of  evaluations  and  thus  the  resources  available  must  be  applied  to  a 
very  narrow  portion  of  the  original  intent. 

•  NSA  and  NIST,  without  separate  funding  earmarked  for  the  NIAP,  have  produced 
a  flexible,  capably  staffed,  although  an  under-funded  product  evaluation  system. 

•  Budgeting  restrictions  have  prevented  the  NIAP  from  developing  education  and 
training  resources  for  IT  system  consumers,  tools  to  support  secure  product  de¬ 
velopment,  and  protection  profiles  for  non-military  applications. 

•  Oversight  of  evaluations  has  been  limited  due  to  stretched  NIAP  budgets  and  a 
shortage  of  qualified  validators.  The  oversight  mechanism  meant  to  ensure  that 
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evaluations  conform  uniformly  to  all  CC  requirements  is  under-funded.  The 
NIAP  has  made  adjustments  in  both  organization  and  resources.  The  current 
number  of  evaluations  stresses  the  limits  of  the  validation  resources  and  the  num¬ 
ber  of  evaluations  continues  to  grow. 

•  Evaluations  take  longer  than  anticipated.  Evaluation  schedules  are  often  extended 
beyond  their  original  plan. 

•  Evaluations  frequently  result  in  modified  products  or  claims.  Most  evaluations 
take  longer  than  anticipated  either  because  the  product  does  not  satisfy  the  initial 
claims,  or  because  the  documentation  is  not  adequate.  The  result  is  that  either  the 
product  or  the  claims  must  be  modified.  This  is  actually  a  good  thing  if  one  as¬ 
cribes  to  the  “truth  in  advertising”  approach,  and  reduction  of  claims  in  marketing 
materials  would  follow.  However,  sufficient  data  gathering  has  not  been  done  to 
adequately  quantify  this  effect.  It  appears  (from  experienced  evaluators  and  vali¬ 
dators  that  the  evaluation  documents  are  the  primary  place  that  claims  are  re¬ 
duced.  Ideally,  the  product  would  be  modified  to  meet  the  claims,  but  it  is  usually 
easier  to  obtain  a  certificate  by  reducing  claims.  The  NIAP  is  working  to  have 
claim  adjustments  documented,  but  this  only  helps  sophisticated  customers  who 
read  it.  When  conformance  to  a  PP  is  cited  or  required,  the  claims  in  the  ST  can¬ 
not  be  reduced  below  those  of  the  PP.  DoD  requirements  for  PP  conformance 
should  be  continued. 

•  Developers  produce  large  amounts  of  data  relevant  to  evaluations  during  their  de¬ 
velopment  and  testing.  Only  a  small  portion  of  this  data  is  provided  to  evaluators 
in  the  form  of  evidence.  Consumers  typically  see  only  the  product’s  evaluation 
certificate,  which  contains  no  vulnerability  information,  and  the  other  information 
available  to  them  is  written  in  precise  evaluation  language,  and  typically  has  little 
information  about  residual  vulnerabilities.  The  Education,  Training  and  Aware¬ 
ness  (ETA)  have  lagged  so  that  consumers  are  generally  not  well  educated  in 
reading  the  evaluation  reports. 

Expectations  and  Expectations-Related  (See  Chapter  5) 

By  far,  this  team  had  the  most  difficult  problem  of  developing  a  coherent  picture  of  ex¬ 
pectations.  The  objective  was  to  gather  as  much  information  as  possible  concerning  vari¬ 
ous  opinions,  experiences  and  expectations  for  NIAP  in  the  context  of  cybersecurity,  in¬ 
formation  assurance  and  product  evaluations.  Every  attempt  was  made  to  solicit  any 
viewpoint  without  judgment  of  its  validity.  Instead,  the  review  effort  collated  the  many 
comments  and  inputs  into  stakeholder  views,  and  organized  the  comments  into  16  topical 
areas  of  coverage.  This  effort  resulted  in  many  observations  that  clearly  may  not  be  as 
important  as  the  same  observation  provided  some  multiple  of  times. 


The  stakeholder  views  were  categorized  as: 

•  Department  of  Defense  (DoD)  -  individuals  in  the  DoD  who  represent  the  as¬ 
sured  information  system  customer  base. 
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•  Federal  Government  (FEDNonDoD)  -  individuals  outside  of  the  DoD  but  in  the 
Federal  government  who  represent  the  customer  base  (such  as  NASA  or  the 
FA  A). 

•  Process  -  individuals  who  are  or  have  been  involved  in  executing  the  current 
NIAP  process,  including  validators  and  lab  personnel,  as  well  as  NIST  and  NSA 
personnel. 

•  Producers  (Large  and  Small)  -  developers  of  IA  or  IA-enabled  software  that 
may  be  subjected  to  evaluation  requirements,  including  large-scale  producers  such 
as  Microsoft,  IBM,  and  Oracle,  as  well  as  small  business  concerns. 

•  Governance  -  individuals  who  are  instrumental  in  making  policy  and  mandating 
requirements  for  their  agencies,  such  as  heads  of  NSA  and  the  NIAP  or  Federal 
agencies. 

•  Defense  Critical  -  individuals  who  are  involved  with  the  operational  capabilities 
of  the  commands  of  the  anned  services.  As  separate  from  branches  of  govern¬ 
ment  such  as  NASA,  FA  A,  etc. 

•  Intelligence  -  individuals  who  are  involved  in  intelligence  gathering  activities. 

The  results  from  the  participants  in  the  collection  process  were  summarized  by  the  team 
into  those  that  clearly  represented  a  stakeholder  expectation  for  product  evaluations 
and/or  NIAP  and  those  that  were  only  an  expression  of  experience  with  no  further  elabo¬ 
ration  of  whether  that  was  intended  to  be  interpreted  as  some  kind  of  expectation. 

The  resulting  topical  areas  used  to  organize  the  inputs  into  simple  observational  groups 
and  expectations  were  as  follows.  The  summary  findings  from  these  detailed  observations 
and  synthesis  of  expectations  are  subordinate  to  each  topical  heading.  In  some  cases,  al¬ 
though  the  topic  was  mentioned  frequently,  no  substantial  finding  was  made  by  this  re¬ 
view. 

Consumer  knowledge  and  understanding  of  evaluations 

•  Consumers  need  a  better  understanding  of  information  assurance  threats  and  pro¬ 
tection  methods,  and  a  basic  understanding  of  NIAP  evaluation  processes  to  inter¬ 
pret  evaluation  results  and  make  infonned  decisions  about  product  suitability  for 
their  needs. 

•  Evaluations  are  often  reported  in  technical  CC  terms  and  do  not  state  in  plain  lan¬ 
guage  what  information  assurance  protection  the  product  provides. 

The  meaning  of  a  product  evaluation  certificate 

•  Evaluation  certificates  in  general  do  not  identify  the  degree  of  security  provided 
by  the  product  or  provide  example  applications  for  which  the  product  is  suitable. 

Protection  profiles 

•  Protection  profiles  covering  core  information  assurance  capabilities  for  general 
use  have  not  been  developed.  A  number  of  protection  profiles  that  address  the 
higher  levels  of  assurance  for  national  security  systems  have  been  developed  by 
NSA  for  use  by  that  community.  Protection  profiles  for  capabilities  that  satisfy 
more  modest  assurance  requirements  have  not  been  developed. 
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Evaluation  personnel  and  evaluation  laboratory  issues 

•  Product  evaluators  come  from  a  variety  of  disciplines  with  varying  levels  of  ex¬ 
pertise.  Although  NIAP  checks  that  evaluation  processes  are  followed  correctly, 
no  process  has  been  established  to  ensure  adequate  training  of  evaluators  and 
validators. 

•  Current  conflict  of  interest  rules,  particularly  those  that  allow  laboratories  to  de¬ 
velop  evidence  and  conduct  evaluations  on  the  same  products,  are  open  to  poten¬ 
tial  abuse. 

Testing  of  products  in  evaluations 

•  Automated  tools  can  help  standardize  evaluation  processes,  perfonn  more  thor¬ 
ough  product  analyses,  and  reduce  evaluation  costs.  No  standard  collection  of 
automated  security  analysis  tools  has  been  developed  or  assembled  to  support 
evaluations. 

•  Both  the  Common  Evaluation  Method  (CEM)  and  protection  profiles  often  omit 
detailed  testing  requirements. 

•  No  automated  review  of  source  code  is  required  for  evaluations  at  EAL-4  and  be¬ 
low.  For  software  products,  the  code  represents  a  complete  technical  specifica¬ 
tion  of  the  product's  functionality,  and  is  much  more  revealing  than  the  other  de¬ 
sign  and  implementation  documentation  that  is  considered  in  evaluations.  Auto¬ 
mated  source  code  review  could  screen  out  many  common  security  flaws  that  cur¬ 
rently  go  undetected. 

Alternate  forms  of  assurance 

•  Alternate  forms  of  assurance  are  not  significant  concerns  for  most  stakeholder 
classes.  Alternate  forms  of  assurance,  however,  may  reduce  costs  and  could  be 
useful  at  lower  evaluation  assurance  levels.  Several  interviewees  believed  that  al¬ 
ternative  assurance  methods  are  needed,  especially  to  reduce  costs  (such  as  a  “CC 
life”).  This  is  the  case  for  organizations  or  situations  that  cannot  afford  to  pay  for 
evaluations,  such  as  many  small  web  applications,  small  businesses,  and  open 
source  software  (OSS)  projects.  Support  for  alternative  assurance  levels  was 
strongest  for  use  in  lower  assurance  evaluations.  Many  believed  that  NIAP 
evaluation  would  be  strengthened  if  the  alternative  assurance  methods  were  used 
to  supplement  the  NIAP  evaluation,  with  SSE,  CMM,  and  CMMI  specifically 
mentioned  as  examples  of  alternative  assurance  methods.  These  assurance  meth¬ 
ods  would  augment  (not  replace)  the  current  assurance  methods. 

Relationship  between  C&A  and  product  evaluation 

•  Certification  and  accreditation  of  systems  was  considered  essential  by  all  stake¬ 
holder  classes,  and  product  evaluation  should  improve  C&A. 

Mutual  recognition,  commercial  viability  and  related  issues 

•  NIAP  has  not  addressed  warranty  or  liability  issues  for  evaluated  products.  No 
legal  or  business-case  analyses  on  who  might  underwrite  warranties  for  evaluated 
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products  was  found,  or  what  effect  warranties  might  have  in  promoting  adoption 
of  evaluated  products. 

•  Mutual  Recognition  is  necessary. 

Research  areas 

•  A  number  of  open  research  problems  remain  unaddressed,  including  assurance 
metrics  and  solutions  to  composability  among  other  security  problems. 

Target  of  evaluation  (TOE)  versus  product  evaluation 

•  A  number  of  products  have  been  evaluated  in  unusual  configurations  and  envi¬ 
ronments  that  do  not  represent  consumers'  general  use.  These  evaluations  do  not 
provide  sufficient  information  to  determine  how  these  products  will  perform  in 
typical  system  configurations  and  normal  use. 

Assurance  maintenance 

•  Evaluations  should  include  both  maintenance  assurance  and  flaw  remediation 
work  packages. 

Cost  and  time  issues 

•  “Evaluation  costs  are  too  high  and  they  take  too  long.”  These  are  common  com¬ 
plaints,  particularly  from  small  businesses.  The  documentation  generated  for 
evaluations  is  partly  responsible.  Although  no  significant  finding  was  made  by 
this  review,  the  comments  and  observations  in  this  topical  area  were  consistent 
with  other  findings  of  the  review. 

NSTISSP-11 

•  Comments  regarding  NSTISSP-11  are  included  for  completeness  in  the  body  of 
the  report;  however,  no  significant  findings  were  made.  However  the  expectations 
are  consistent  with  other  findings  of  the  report;  such  as  the  need  for  protection 
profiles  and  issues  related  to  perceived  and  actual  costs  incurred  to  comply. 

Critical  infrastructure 

•  No  finding  was  made  in  this  category.  However,  an  expectation  expressed  by 
stakeholders  from  all  classes  was  that  the  Critical  Infrastructure  Protection  (CIP) 
community  should  be  brought  under  the  national  security  mandates.  Most  gov¬ 
ernment  departments  and  agencies  that  are  part  of  CIP  are  already  under  the 
FISMA  mandates.  Including  CIP  under  product  evaluation  and  CC  mandates  may 
create  an  undue  burden  of  cost. 

Nefarious  and  malicious  behavior  in  code 

•  Although  there  was  little  input  on  this  subject,  malicious  code  and  backdoor  ac¬ 
cess  paths  inserted  during  development  have  to  be  considered  in  any  assurance 
arguments.  Many  of  the  interviewees  felt  uncomfortable  discussing  this  area  and 
there  was  little  written  input.  Nevertheless  tools  for  specification  analysis,  exam¬ 
ining  code  and  product  execution  and  manage  configurations  are  often  encoun- 
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tered  in  the  literature  as  necessary  for  product  evaluations  if  both  security  effec¬ 
tiveness  and  affordability  is  to  be  achieved. 

Comments  concerning  NIST 

•  No  finding  is  provided  for  this  topic,  but  it  is  noted  by  stakeholders  that  NIST  in¬ 
volvement  is  minimal  and  decreasing. 

For  further  elaboration  of  these  findings,  Section  5.4  of  Chapter  5  summarizes  the  756 
recorded  comments  from  the  collection  of  interviews,  forum  discussions,  and  other  con¬ 
tributed  input.  "Issues"  in  Chapter  5  represent  the  principal  concerns  raised  by  interview¬ 
ees,  Forum  participants,  and  other  contributors.  "Expectations"  represent  recommenda¬ 
tions  expressed  by  these  sources. 

Options  for  the  Way  Ahead 

Analysis  of  the  issues,  findings,  and  conclusions  led  to  the  identification  of  six  options 
for  the  NIAP,  in  increasing  magnitude  of  change. 

Option  1:  Eliminate  the  NIAP  and  product  evaluations 

Shift  virtually  all  of  the  responsibility  for  information  system  security  to  system-level 
Certification  &  Accreditation  (C&A).  System  C&A  is  necessary  anyway,  even  with 
evaluated  products,  but  it  would  no  longer  benefit  from  the  NIAP’s  vetting  of  component 
products.  C&A  of  separate  systems  would  duplicate  the  effort  of  vetting  common  com¬ 
ponents. 

Option  2:  Continue  the  NIAP  in  its  current  form  (reduced  from  the  original  intent) 

Continue  the  informal  partnership  between  NIST  and  NSA,  and  continue  to  monitor 
product  evaluations  and  participate  in  Mutual  Recognition  Arrangement  and  Common 
Criteria  (CC)  improvement  activities.  Additional  personnel  would  be  needed  to  handle 
the  growth  of  evaluations.  Because  of  budget  constraints,  however,  current  NIAP  activi¬ 
ties  do  not  include  many  of  the  research  and  development  objectives  outlined  in  its  origi¬ 
nal  charter. 

Option  3:  Restore  the  NIAP  to  the  original  intent  of  the  Letter  of  Partnership  be¬ 
tween  NSA  and  NIST 

Restore  the  NIAP  to  the  full  functioning  envisioned  when  the  partnership  was  first  estab¬ 
lished  in  1998.  Bolster  the  NIST-NSA  memorandum  of  agreement  with  a  more  formal 
charter  and  assignment  of  responsibilities,  and  direct  the  agencies  to  provide  adequate 
funding.  Consider  additional  partners,  for  example  a  component  from  DHS.  This  option 
addresses  many  issues  raised  about  current  NIAP  operations,  but  it  does  not  address 
changes  in  the  cybersecurity  environment  that  have  occurred  since  its  inception. 

Option  4:  Modernize  the  approach  to  cybersecurity  evaluations  to  reflect  changes  in 
the  environment  since  its  creation  in  1998 

In  addition  to  restoring  the  NIAP  as  described  in  Option  3,  provide  more  stable  funding 
in  the  form  of  a  national  budget  line  item,  and  strengthen  oversight  of  NIAP  activities. 
Then  update  the  NIAP’s  processes  to  address  changes  in  the  cybersecurity  environment. 
For  example,  many  software  vulnerabilities  are  caused  by  a  relatively  small  set  of  com¬ 
mon  implementation  errors,  and  many  of  these  errors  can  be  detected  or  countered  by 
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tools.  Requiring  use  of  approved  software  tools  would  reduce  product  evaluation  costs 
and  improve  effectiveness.  Vulnerability  testing  would  be  included  in  all  evaluations,  not 
just  in  those  of  high-end  products. 

Option  5:  Integrated  approach  to  cybersecurity  evaluations 

Extend  Option  4  to  integrate  product  evaluations  into  the  larger  context  of  information 
system  security.  Rather  than  merely  testing  finished  products  in  limited  contexts,  evalua¬ 
tions  need  insight  into  a  product’s  original  security  objectives  and  assurance  aspects  of 
the  development  processes  used  to  produce  it.  Complex  products  have  wide  ranges  of 
possible  configurations,  only  small  fraction  of  which  are  subjected  to  evaluation.  Appli¬ 
cations  are  similarly  wide  ranging,  presenting  a  huge  variety  of  possible  environments 
and  often  requiring  extensive  configuration  adjustments.  Current  product  evaluations  do 
not  provide  sufficient  information  for  C&A  analyses.  Research  is  needed  to  fully  inte¬ 
grate  these  cybersecurity  components. 

Option  6:  Forward  looking  approach  to  cybersecurity  evaluations  (new paradigm) 

Move  security  evaluations  to  a  new  paradigm.  This  is  not  an  incremental  change  that  fol¬ 
lows  the  progression  already  described,  but  a  completely  new  way  of  thinking  about  the 
problem  of  cybersecurity.  While  it  is  not  possible  to  describe  this  option  and  associated 
resource  requirements  in  actionable  detail  without  further  study,  several  aspects  of  this 
option  are  clear  at  this  time.  The  new  paradigm  must  address  future  risks  and  vulnerabili¬ 
ties  to  systems.  It  must  ensure  that  the  security  of  a  system  is  greater  than  the  sum  of  its 
parts.  It  must  also  address  issues  related  to  changes  in  system  ownership,  rapid  changes 
in  system  composition,  changes  in  data  ownership  and  location,  changes  in  user  exper¬ 
tise,  and  increasingly  complex  systems.  In  other  words,  the  new  paradigm  must  be  as 
dynamic  as  the  environment  within  which  it  must  work. 

Summary  of  Review  Recommendations 

A  number  of  actions  are  recommended  to  converge  on  a  responsive  approach  to  product 
evaluations  within  an  overall  context  of  cybersecurity  and  infonnation  assurance,  which 
is  expected  to  be  derived  from  a  combination  of  Options  3  to  5,  with  Option  6  as  a  goal. 
Although  the  reviews  recommends  Option  3  as  a  minimum  to  resource  and  restore  prod¬ 
uct  evaluations  to  the  original  intent  of  NIAP,  this  first  requires  adequate  resources  to  ac¬ 
complish  Option  2,  based  on  the  rapidly  increasing  numbers  of  evaluations  in  the  pipeline 
and  the  limited  resource  capacities  of  the  NIAP  partners  to  conduct  the  minimum  of  ac¬ 
tivities  required.  Although  Options  4  and  5  may  appear  similar,  they  are  in  fact  distinct. 
Option  4  is  intended  to  maximize  the  benefits  of  evaluations  against  standards;  whereas 
Option  5  is  intended  to  add  capabilities  that  would  enable  product  evaluations  against 
standards  to  be  integrated  with  other  processes  and  their  assessment  and  evaluation  meth¬ 
ods;  such  as  Certification  and  Accreditation  (C&A)  and  Test  and  Evaluation  (T&E)  of 
systems  and  their  operating  environments  and  specific  configurations.  This  review  con¬ 
cludes  that  such  an  integration  will  a  significant  challenge  and  will  likely  require  signifi¬ 
cant  investment  that  may  not  be  provided  to  NIAP  but  represents  a  point  of  interface  for 
NIAP  and  the  cybersecurity  operating  environment.  Thus,  at  a  minimum  it  could  extend  it 
charter  to  include  such  interactions  at  a  minimum  as  part  of  the  larger  cybersecu¬ 
rity/information  assurance  issues. 
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It  is  recommended  that  DoD  continue  to: 


•  Develop  Protection  Profiles  (PPs)  for  DoD  and  National  Security  applications. 

•  Require  conformance  to  PPs,  where  they  are  available. 

•  Require  vulnerability  testing  for  products  at  the  lower  evaluation  levels. 

•  Include  aspects  of  flaw  remediation  and  assurance  maintenance  in  PPs. 

Specifically,  it  is  also  recommended  that  DoD: 

•  Require  product  evaluations  to  include  maintenance  assurance  and  flaw  reme¬ 
diation  in  accordance  with  the  Common  Criteria. 

•  Support  the  full  integration  of  product  evaluation  and  C&A  processes. 

•  Support  the  development  and  use  of  software  tools  for  vulnerability  analyses. 

•  Participate  in  an  annual  assessment  and  review  of  the  nation’s  cybersecurity 
posture. 

•  Support  the  development  of  a  lower  cost,  alternate  form  of  assurance  for 
products  at  lower  assurance  levels. 

It  is  recommended  that  DHS: 

•  Collaborate  with  DoD  on  the  initiatives  above  and  support  extensions  of  these 
efforts  to  all  federal  and  commercial  IA  products. 

•  Support  vulnerability  testing  of  all  products  undergoing  evaluation. 

•  Support  the  development  of  a  set  of  core  functionality  protection  profiles  for 
use  by  federal  departments  and  agencies,  by  critical  infrastructure  compo¬ 
nents,  and  by  the  commercial  sector. 

•  Support  the  use  of  core  protection  profiles,  where  applicable,  to  give  product 
buyers  confidence  in  the  product’s  security  functionality  and  suitability  for 
use. 

•  Support  the  full  integration  product  evaluation  and  C&A  processes  for  federal 
departments  and  agencies  and  critical  infrastructure  components. 


Report  Organization 

This  report  provides  a  large  amount  of  detailed  information.  Some  suggestions  for  find¬ 
ing  material  of  specific  interest  are  offered  at  the  end  of  Chapter  1,  following  the  descrip¬ 
tion  of  the  report’s  organization.  Detailed  page  navigation  for  each  finding  and  recom¬ 
mendation  in  each  section  can  be  found  in  Annex  I. 
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1. 


Introduction 


1 . 1  Background 

The  National  Information  Assurance  Partnership  (NIAP)  is  a  joint  effort  of  the  National 
Institute  of  Standards  and  Technology  (NIST)  and  the  National  Security  Agency  (NSA)  to 
provide  technical  leadership  in  the  research  and  development  of  security-related  informa¬ 
tion  technology  test  methods  and  assurance  techniques.  To  quote  from  the  letter  that  es¬ 
tablished  the  partnership  [NIST/NSA1997] 1 2 : 

Consumers,  in  both  private  and  public  sectors,  need  confidence  and  assurance  in 
the  products  they  use  to  secure  valuable  infonnation.  That  confidence  is  bolstered 
when  their  products  have  been  evaluated,  tested,  and  certified  by  independent  or¬ 
ganizations.  As  security  products  change  to  stay  ahead  of  evolving  threats,  so 
must  the  tests,  methods,  and  metrics  used  to  evaluate  them.  The  NIAP  will  em¬ 
ploy  the  latest  techniques  to  develop  specification-based  tests,  methods,  and  tools 
so  that  testing  laboratories  and  certificate  issuing  organizations — as  well  as  con¬ 
sumers  and  producers  of  information  technology  products — will  have  objective 
measures  for  evaluating  quality  and  security. 

In  addition  to  boosting  consumer  confidence  in  infonnation  security  products,  a  second 
goal  of  the  NIAP  is  to  enhance  the  U.S.’s  ability  to  gain  international  recognition  and  ac¬ 
ceptance  for  U.S.  products  that  help  ensure  the  security  of  information  technology  sys¬ 
tems  and  networks.  The  Terms  of  Reference  that  forms  the  basis  for  the  working  relation¬ 
ship  between  the  two  organizations  is  found  in  [NIST/NSA1998]. 

Action/Recommendation  4-4  of  The  National  Strategy  to  Secure  Cyberspace  [WH2003] 
requires  the  Federal  Government  to  conduct  a  comprehensive  review  of  the  NIAP  to  de¬ 
termine  the  extent  to  which  it  is  adequately  addressing  the  continuing  problem  of  security 
flaws  in  commercial  software  products.  This  review  will  include  lessons-learned  from 
implementation  of  the  DoD’s  July  2002  policy  requiring  the  acquisition  of  products  re¬ 
viewed  under  the  NIAP  or  similar  evaluation  processes.  The  DoD  and  the  Department  of 
Homeland  Security  (DHS)  cooperated  on  the  review.  This  report  presents  the  results  of 
that  review.  IDA  conducted  this  review  in  the  broadest  of  terms  based  on  an  initial  set  of 
questions  provided  by  the  Homeland  Security  Council  (HSC)  and  after  subsequent  con- 


1  The  complete  text  of  the  letter  of  agreement  is  provide  in  Annex  G. 

2 

In  January  2000,  the  Committee  on  National  Security  Systems  (CNSS),  formerly  the  National  Security  Telecommunications  and 
Information  Systems  Security  Committee  (NSTISSC),  issued  National  Information  Assurance  Acquisition  Policy  (NSTISSP  No.  1 1). 
That  policy  directs,  “by  1  July  2002,  the  acquisition  of  all  commercial  off-the-shelf  (COTS)  Information  assurance  (IA)  and  IA- 
enabled  information  technology  (IT)  products  shall  be  limited  only  to  those  which  have  been  evaluated  and  validated  in  accordance 
with  criteria,  schemes,  or  programs  of  the  Common  Criteria,  the  National  Information  Assurance  Partnership  (NIAP)  evaluation  and 
validation  program,  and  the  Federal  Information  Processing  Standards  (FIPS)  validation  program.”  [NST2003] 
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versations  with  DoD  (ASD/NII/DIAP)  and  DHS  (NCSD)  organizations  cognizant  of 
oversight  issues  surrounding  product  evaluations. 

1 . 2  Security  in  Cyberspace 

The  National  Strategy  to  Secure  Cyberspace  [WH2003],  issued  in  February  2003,  argues 
that  the  information  technology  revolution  has  quietly  changed  the  way  business  and 
government  operate.  The  U.S.  increasingly  relies  on  an  interdependent  network  of  infor¬ 
mation  technology  infrastructures  called  cyberspace.  The  security  of  cyberspace  is  es¬ 
sential  to  our  economy  and  national  security,  as  numerous  recent  cyber  attacks  have  dem¬ 
onstrated.  Testing  and  certifying  that  commercial  products  and  systems  are  free  of 
known  and  applicable  security  vulnerabilities  and  weaknesses  are  an  integral  part  of  the 
strategy  to  secure  cyberspace.  Freedom  from  all  security  vulnerabilities  and  weakness  is 
an  impossible  task  with  today’s  technology;  however  relative  freedom  from  known  and 
applicable  security  vulnerabilities  is  achievable.  The  need  has  been  growing  since  1970, 
often  in  spurts,  with  responses  that  vary  to  meet  specific  situations.  Annex  E  provides  an 
historical  look  at  the  timeline  of  events  leading  to  the  development  of  the  NIAP  and  the 
associated  National  Policy  regarding  Information  Assurance  (IA).  This  array  of  policy 
covering  cybersecurity,  IA  and  NIAP  is  a  complex  and  exhaustive  response  to  growing 
cybersecurity  concerns  with  well-intentioned  approaches  to  cybersecurity  but  also  gener¬ 
ating  an  overlapping  array  of  requirements  and  mandates.  These  will  be  discussed  at 
length  in  Chapter  3 . 

1 . 3  NIAP  Scope 

The  long-tenn  goal  of  the  NIAP  is  to  help  increase  the  level  of  trust  consumers  have  in 
their  information  systems  and  networks  through  the  use  of  cost-effective  security  testing, 
evaluation,  and  validation  programs.  In  meeting  this  goal,  the  NIAP  seeks  to: 

•  Promote  the  development  and  use  of  evaluated  IT  products  and  systems; 

•  Champion  the  development  and  use  of  national  and  international  standards  for 
IT  security; 

•  Foster  research  and  development  in  IT  security  requirements  definition,  test 
methods,  tools,  techniques,  and  assurance  metrics; 

•  Support  a  framework  for  international  recognition  and  acceptance  of  IT  secu¬ 
rity  testing  and  evaluation  results;  and 

•  Facilitate  the  development  and  growth  of  a  commercial  security  testing  indus¬ 
try  within  the  U.S. 

The  NIAP  has  made  progress  in  all  of  these  with  the  exception  of  the  third  point,  for  at 
least  the  defense  industry.  Additionally,  the  NIAP  has  made  adaptations  or  been  forced 
to  adjust  activity  emphasis  to  accommodate  the  changing  pressures  brought  about  by  the 
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complexities  of  cybersecurity  while  remaining  within  the  bounds  of  a  letter  of  partner¬ 
ship  and  funding  constraints. 

1 . 4  Evolution  of  NIAP 

For  over  two  decades,  NIST  and  NSA  have  promoted  security  in  commercial  off-the- 
shelf  Infonnation  Technology  (IT)  products.  These  efforts  focused  initially  on  the  gov¬ 
ernment-sponsored  Trusted  Computer  System  Evaluation  Criteria  (TCSEC).  Several  fac¬ 
tors  during  the  past  decade  influenced  the  hannonizing  of  these  evaluation  criteria  lead¬ 
ing  up  to  the  internationally  accepted  and  standards-based  common  criteria,  ISO  Interna¬ 
tional  Standard  15408. 

These  factors  included: 

1 .  Development  of  similar  IT  security  evaluations  criteria  by  other  nations; 

2.  Globalization  of  the  IT  product  market; 

3.  Inclusion  of  security  into  middleware,  applications,  and  network  devices; 
and 

4.  Cost  and  time  span  of  evaluations. 

At  the  same  time  that  the  Common  Criteria  (CC)  standard  was  being  developed,  NSA 
began  the  transition  of  its  Trusted  Product  Evaluation  Program  to  the  private  sector.  This 
transition  continues  today  under  the  NIAP  Common  Criteria  Evaluation  and  Validation 
Scheme  for  IT  Security  (CCEVS). 

The  NIAP  was  initiated  by  an  August  1997  agreement  between  NIST  and  NSA 
[NIST/NSA1997].  Almost  a  year  later,  NSA  and  NIST  signed  a  Terms  of  Reference 
document  [NIST/NSA1998]  that  included  the  statement  that  the  agreement  can  be  modi¬ 
fied,  rescinded  or  terminated  unilaterally.  The  Terms  of  Reference  gives  each  organiza¬ 
tion  an  equal  voice  in  all  aspects  of  the  NIAP  decision-making,  including  the  selection  of 
the  NIAP  projects,  allocation  of  the  NIAP  resources,  oversight  of  contractor  support,  and 
technical  direction.  NIST  and  NSA  designate  Management  Representatives  (MRs)  to 
provide  guidance,  direction,  and  priorities  to  the  NIAP.  The  MRs  must  jointly  agree  on 
activities  designated  as  the  NIAP  projects.  The  Tenns  of  Reference  are  silent  on  the  sub¬ 
ject  of  funding  except  to  say  that 

Recognizing  that  NIST  and  NSA  will  not  have  equality  in  the  amount  of  discre¬ 
tionary  resources  applicable  to  NIAP,  each  organization  will  first  look  internally 
for  resources  and/or  contractual  vehicles  to  accomplish  a  given  task.  If  necessary, 
it  is  agreed  that  one  organization  may  transfer  funds  to  the  other  organization  for 
the  purpose  of  supporting  a  specific  the  NIAP  tasking  thought  separate  imple¬ 
menting  agreements  under  the  Economy  Act  (31  U.S.C.  1535). 

1 . 5  Disclaimer 

The  analysis  approach  utilized  analysts  from  a  number  of  backgrounds  and  experiences  in 
a  team  approach  (a  total  of  11  analysts  were  utilized).  This  report  documents  their  con¬ 
sensus.  Several  members  have  had  and  continue  to  have  experience  with  the  CCEVS 
validation  program  (2  are  currently  validators  under  the  CCEVS  program).  While  the  use 
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of  their  expertise  and  understanding  were  valuable  resources  to  the  study,  their  inputs  did 
not  shape  or  determine  the  contents  of  this  report.  Their  inputs  were  treated  as  valuable 
and  experiential  based,  but  identical  to  other  inputs  in  reaching  the  consensus  process. 

1 . 6  Report  Organization 

The  report  has  been  developed  for  a  variety  of  audiences  ranging  from  experts  in  cyber¬ 
security  and  product  evaluation  to  top-level  decision  makers  only  briefly  familiar  with 
these  concepts.  As  a  result,  the  material  is  divided  into  sections  that  contain  great  detail 
and  sections  that  summarize  issues.  The  breakdown  of  the  report  is  given  below: 

•  Chapter  1  covers  the  background,  introduction  and  administrative  items. 

•  Chapter  2  covers  the  scope  and  expectations  of  the  study. 

•  Chapter  3  discusses  the  underlying  policy  basis  in  detail  for  the  NIAP  and  the  cur¬ 
rent  state  of  policy  implementation. 

•  Chapter  4  describes  the  NIAP  and  how  it  has  evolved  from  its  inception  to  its  cur¬ 
rent  organization  and  responsibilities. 

•  Chapter  5  summarizes  the  expectations  of  the  stakeholders  that  were  gleaned  from 
interviews,  solicited  inputs  and  literature  search. 

•  Chapter  6  integrates  the  findings  of  Chapter  3,  4  and  5  and  provides  overall  areas 
of  concerns  and  approaches  to  some  solutions,  including  the  tradeoffs  necessary 
to  synthesize  programs. 

•  Chapter  7  builds  on  Chapter  6  by  providing  approaches  and  options  for  the 
courses  of  action  concerning  the  use  of  product  evaluation  in  an  overall  cyberse¬ 
curity  framework. 

•  Chapter  8  provides  detailed  actions  necessary  to  implement  the  options  of  Chapter 
7. 

Detailed  information,  or  information  too  voluminous  to  include  in  the  body  of  the  report 
are  provided  in  Annexes  as  follows: 

•  Annex  A  contains  the  References  and  Bibliography. 

•  Annex  B  lists  Acronyms. 

•  Annex  C  contains  the  Glossary. 

•  Annex  D  contains  further  policy  information. 

•  Annex  E  provides  NIAP  historical  data. 

•  Annex  F  discusses  software  tools  for  security  analysis  and  proactive  defense. 

•  Annex  G  contains  the  letter  of  agreement  that  established  the  NIAP. 

•  Annex  H  is  a  discussion  of  Alternate  Forms  of  Assurance. 

•  Annex  I  is  an  index  of  the  tags  used  in  the  document  to  describe  assumptions,  ex¬ 
pectations,  observations,  findings,  and  recommendations. 
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•  Annex  J  provides  an  estimate  of  the  costs  to  implement  individual  options. 

The  complexity  of  the  subject  and  the  depth  of  the  analysis  may  dictate  how  this  report  is 
best  used.  For  example,  those  who  wish  to  know  what  options  are  available  and  what 
steps  are  required  to  implement  them  may  wish  to  proceed  directly  to  Chapters  7  and  8. 
Footnotes  and  cross  references  will  eventually  get  you  to  source  data  of  interest.  The  in¬ 
dividual  who  wants  a  level  of  detail  beyond  the  executive  summary  may  wish  to  read 
chapters  1  and  2,  and  then  skip  to  chapter  6.  Chapter  6  is  the  roll-up  of  a  number  of  de¬ 
tails,  with  sufficient  cross-reference  to  allow  back  tracing  to  the  text  in  chapters  3-5. 
Most  of  the  supporting  data  (although  not  all)  has  been  provided  in  annexes  to  improve 
the  flow. 
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2 .  Scope  and  Approach  of  Review 


The  National  Strategy  to  Secure  Cyberspace  [WH2003]  required  the  Federal  Government 
to  conduct  a  comprehensive  review  of  the  NIAP.  The  Review  Tasking  section  below  de¬ 
scribes  the  scope  and  expectations  of  the  task  statement  provided  to  the  Institute  for  De¬ 
fense  Analyses.  Following  sections  discuss  the  terminology  used  in  the  report,  identify 
communities  of  interest,  and  describe  the  review  methodology. 

2 . 1  Review  of  Tasking 

For  this  review,  DoD  and  the  DHS  tasked  the  IDA  on  behalf  of  the  Federal  government 
to  consider  results  of  current  policy  and  practices,  the  general  efficacy  and  adequacy  of 
current  capabilities,  and  the  subsequent  affordability  and  viability  of  expanding  the  NIAP 
program  to  all  federal  agencies.  The  direction  was: 

1.  Characterize  the  NIAP  intent  and  future  expectations,  conduct  fact  finding,  and 
develop  issues; 

2.  Assess  impacts  of  selected  issues  and  generate  alternatives  and  options  to  address 
these  issues; 

3.  Analyze  selected  issues  and  options;  and 

4.  Recommend  option(s)  and  an  implementation  roadmap. 

The  scope  of  this  task  covers  government-wide  issues,  although  particular  emphasis  is 
placed  on  DoD  and  DHS  issues  and  concerns.  This  scope  is  broader  than  NIAP  as  it  cur¬ 
rently  exists.  Findings  and  recommendations,  therefore,  may  apply  to  other  aspects  of  the 
cybersecurity  problem  not  currently  addressed  by  NIAP.  The  options  explored  lead  to  an 
integrated  approach  to  a  product-evaluation  capability  without  prejudice  to  who  has  re¬ 
sponsibility  for  execution.  The  complexity  of  the  issues,  limited  cost/benefit  data  and 
options  made  it  out-of-scope  to  provide  detailed  cost  estimates  after  developing  options. 
However,  the  NSA  and  NIST  provided  estimates  of  cost  to  implement  several  of  the  op¬ 
tions  after  reviewing  with  IDA  for  clarification  of  possible  ways  ahead.  However,  further 
effort  is  required  to  document  and  fully  explore  a  rigorous  analysis  of  alternatives  or 
business  case  analysis  for  product  evaluations  within  the  surrounding  context  of  cyberse¬ 
curity  and  I  A. 

2 . 2  Methodology 

The  following  describes  the  specific  review  approach. 

The  review  team  took  a  three-pronged  approach  to  reviewing  the  NIAP  (See  Figure  1). 
Since  Federal  and  agency  policies  ultimately  dictate  requirements,  one  team  explored 
policy  to  detennine  the  need,  what  the  NIAP  must  be.  The  policy  review  is  documented 
in  Chapter  3.  A  second  team  reverse-engineered  the  current  NIAP  process  in  order  to 
make  observations  about  practice,  what  the  NIAP  currently  is.  The  process  review  is 
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documented  in  Chapter  4.  A  third  team  delved  into  the  expectations  of  various  NIAP 
stakeholder  groups  in  order  to  find  out  what  users  expect  and  need.  This  was  done 
through  interviews,  an  open  forum,  and  a  notice  in  the  Federal  Register  soliciting  com¬ 
ments  on  the  NIAR  The  expectations  review  is  documented  in  Chapter  5. 


What  do  users  expect  and 
need? 


What  requirements  does  NIAP 
meet  and  how  are  they  met? 


What  requirements  are  derivable  from  DoD/DHS/U.S.  Gov¬ 
ernment  (USG)  documents 

(Legal,  Regulatory,  Policy) 

Figure  1.  Three-Pronged  Approach 

Several  common  themes  arose  in  each  area  of  review.  Results  from  the  three  areas  are 
integrated  into  a  complete  perspective  in  Chapter  6.  This  combines  the  common  findings 
and  attempts  to  resolve  apparent  contradictions  between  requirements,  process,  and  ex¬ 
pectations.  The  teams  then  identified  options  for  closing  the  gaps  and  analyzed  each  op¬ 
tion  for  resources,  feasibility,  and  time  scale.  Chapter  7  describes  six  options,  along  with 
their  pros  and  cons.  Chapter  8  is  a  roadmap  of  near-,  mid-,  and  long-term  actions  to  ac¬ 
complish  each  option. 

The  scope  of  the  review  included  the  environment  from  which  needs  and  expectations 
arise  and  into  which  NIAP  performance  results  deliver  benefits  or  consequences.  The 
method  above  primarily  deals  with  a  characterization  of  the  situation  both  external  to 
NIAP  and  within  the  partnership.  The  assessment  of  issues  and  development  of  options 
and  recommendations  depends  also  on  obtaining  adequate  data  and  information  to  indi¬ 
cate  how  well  NIAP  is  performing  and  relative  to  some  baseline.  One  baseline  is  relative 
to  itself.  Another  is  relative  to  externally  imposed  metrics  and  measures  of  value.  The  fol¬ 
lowing  discussion  is  meant  to  provide  a  perspective  on  the  notional  data  and  information 
the  review  tried  to  obtain  in  order  to  provide  a  quantitative  picture  of  NIAP  -  its  value 
and  its  costs.  Such  data  was  sparse  and  the  review  was  not  able  to  acquire  sufficient  quan¬ 
titative  data  to  support  strong  cost-benefit  arguments.  However,  the  following  back¬ 
ground  provides  the  basis  that  the  review  assumed  could  be  characterized  when  it  initi¬ 
ated  its  collection  of  data  and  assessment  of  performance  and  cost. 
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2 . 3  Background  -  Cybersecurity  Landscape 

Sufficient  anecdotal  evidence  prior  to  the  review  and  observations  during  the  conduct  of 
the  review  indicate  the  NIAP  role  is  poorly  understood.  Also  heard  expressed  were  de¬ 
sires  for  the  NIAP  to  be  and  do  more  than  it  actually  does.  Significantly,  these  percep¬ 
tions  set  the  stage  for  what  the  review  affirmed.  Namely,  that  cybersecurity  issues  gener¬ 
ally  were  being  confused  with  NIAP  -  not  by  the  NIAP  but  certainly  by  users  and  in 
some  cases  by  those  involved  in  related  aspects  of  NIAP.  Before  examining  the  NIAP 
and  its  role,  an  analogous  example  is  provided  to  assist  the  reader.  This  analogy  was  fre¬ 
quently  encountered  during  interviews  with  stakeholders  and  is  only  to  establish  (al¬ 
though  imperfect)  analogy  for  separating  NIAP  roles/responsibilities  from  other  cyberse¬ 
curity  roles  and  responsibilities. 

The  anecdotal  evidence  prior  and  the  subsequent  expectations  comments  affirmed  by  the 
review  that  confusion  and  discussion  of  NIAP  role  and  scope  in  the  cybersecurity  context 
was  shaped  by  misunderstanding  spanning  five  basic  areas  that  set  scope  and  boundaries 
for  cybersecurity  capabilities.  These  areas  of  scope  confusion  were  generally  the  follow¬ 
ing  and  the  reader  of  the  report  should  keep  this  model  in  mind  when  reading  the  details 
of  the  report. 

•  Characterization  of  products  claiming  security  features,  functions  or  properties 
(input  to  NIAP); 

•  Characterization  of  evaluation  processes  for  products  with  security  features,  func¬ 
tions  or  properties  (NIAP  value-adding  activities); 

•  Characterization  of  implemented  systems  incorporating  evaluated  products  and 
non-evaluated  products  (use  of  NIAP  evaluations); 

•  Characterization  of  operational  outcomes  or  effectiveness  of  systems  with  security 
expectations  from  using  evaluated  products  (outcome/benefit  measurement  versus 
input/output  measurement);  and 

•  Characterization  of  equipment,  software,  and  expertise  to  perform  evaluations  and 
their  limitations  in  determining  security  features,  functions,  or  properties  of  prod¬ 
ucts  against  standards  and  in  operational  use  (evaluation  tools,  techniques  and  in¬ 
frastructure). 

2.3.1  Physical  Security  Analogy 

Consider  the  security  of  a  house.  In  this  example,  the  house  is  the  system  and  the 
neighborhood  is  the  environment.  Security  is  provided  by  a  set  of  products  such  as  door 
locks,  window  locks,  bars  on  the  windows,  broken  window  detectors,  motion  sensors, 
and  alarms.  There  are  different  types  of  locks  and  sensors,  which  would  correspond  to  the 
algorithms  in  computer  security  systems.  A  five-tumbler  lock  may  take  more  time  to  pick 
or  force  than  a  three-tumbler  lock.  Effective  placement  of  these  products  depends  on  the 
layout  of  the  house.  In  addition,  effectiveness  depends  on,  the  algorithm  (5  tumbler  locks 
versus  3  tumbler)  and  how  they  are  used:  if  someone  does  not  set  the  alarm  system, 
forced  entry  may  not  be  detected  until  too  late.  In  the  end,  alarms  are  tested,  sensors  are 
tested,  and  scenarios  of  break-ins  are  run  against  the  system  to  be  sure  the  risks  are  un¬ 
derstood. 
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Note  that  security  is  not  absolute.  The  amount  of  security  in  an  information  system  is 
restricted  by  the  value  of  the  data:  someone  does  not  pay  more  for  the  security  devices 
than  the  data  being  protected  is  worth.  Also,  security  devices  tend  to  make  systems 
harder  to  use.  Banks  have  lots  of  money  and  pay  a  lot  for  such  things  as  vaults  and  armed 
guards;  even  so,  they  are  occasionally  robbed.  The  goods  in  a  house  typically  are  of 
much  less  value  than  the  money  in  a  bank.  Most  homes  do  not  have  alarm  systems. 
These  systems  add  cost  to  install  and  to  provide  monitoring;  they  also  make  entering  and 
leaving  the  house  more  difficult  since  the  alann  must  be  turned  off  and  turned  back  on. 

Information  security  products  may  implement  some  algorithm,  which  might  differentiate 
the  strength  and  usefulness  of  similar  products.  For  example,  while  many  operating  sys¬ 
tems  authenticate  users,  some  do  so  using  passwords  and  others  using  smart  cards.  In  a 
house,  while  all  locks  protect  against  unauthorized  entry,  some  use  a  key  to  open,  others  a 
combination,  and  still  others  a  garage  door  opener.  The  number  of  pins  may  further  clas¬ 
sify  those  that  use  keys. 

To  evaluate  the  security  of  a  house,  one  should  start  with  the  overall  layout  of  the  house 
and  the  placement  of  the  various  security  products  to  see  if  there  are  unguarded  entry 
points  or  mismatches  between  products.  Identified  weaknesses  should  be  tested  to  see  if 
they  are  exploitable.  Also,  an  overall  attack  should  be  tried  to  determine  if  there  are  un¬ 
expected  weaknesses.  If  the  same  model  of  lock  is  used  on  each  door,  however,  one  need 
not  try  to  pick  each  of  them:  one  is  enough.  Furthennore,  if  some  laboratory  has  already 
tested  the  lock  and  determined  that  it  properly  incorporates  some  number  of  pins  and  that 
its  case  has  a  specified  hardness,  the  strength  of  the  lock  does  not  need  to  be  tested  at  all 
at  the  house,  but  rather  just  that  it  has  been  properly  installed. 

Laboratory  testing  of  individual  security  products  such  as  locks  and  motion  sensors  cor¬ 
responds  to  NIAP’s  product  evaluation  function.  Evaluating  the  security  of  a  house  that 
incorporates  those  products  corresponds  to  another  important  function,  Certification  and 
Accreditation  (C&A).  After  ensuring  that  the  devices  are  installed  and  configured  prop¬ 
erly,  C&A  can  rely  on  the  laboratory’s  protection  findings. 

2.3.2  Cybersecurity  Problem  Decomposition 

Figure  2  articulates  a  security  evaluation  framework  in  which  the  NIAP  role,  both  current 
and  as  it  may  evolve,  can  be  understood,  and  that  can  be  used  to  understand  the  findings 
and  conclusions  of  this  review.  Furthennore,  this  framework  should  help  the  reader  of 
this  report  to  separate  “what  the  NIAP  is”,  “what  the  NIAP  does”,  and  “what  may  be  ex¬ 
pected  of  the  NIAP”.  It  can  also  be  used  to  answer  the  question  of  what  interfaces  does 
the  NIAP  have  with  other  security  evaluation  processes.  Figure  2  illustrates  the  frame¬ 
work  as  an  “onion-skin”  of  successively  increasing  statements  of  security  worthiness. 
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The  environment  consists  of  a  changing  set  of  conditions, 
policies,  and  other  factors  unknown  at  the  time  of 
implementation  but  realized  during  use  or  consumption 
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Figure  2.  Framework  of  cybersecurity  relationships 

Information  assurance  or  IA-enabled  products  contain  functionality  that  supports  confi¬ 
dentiality,  integrity,  authorization,  availability,  and  other  I A  functions.  However,  prod¬ 
ucts  that  do  not  fit  this  category  may  very  well  have  an  impact  on  the  security  of  systems. 
Those  products,  such  as  word  processors,  spreadsheets,  financial  utilities,  etc.,  are  not 
normally  thought  of  as  IA  or  IA-enabled  should  none-the-less  be  candidates  for  vulner¬ 
ability  testing,  which  is  discussed  later.  Ultimately,  an  organization  should  be  concerned 
with  its  entire  information  infrastructure.  This  infrastructure  consists  of  a  system  (the 
house  in  the  analogy)  of  many  nodes  connected  by  physical  and  wireless  networks.  The 
system  operates  in  an  environment  that  is  partially  controlled  and  partially  uncontrolled 
(the  neighborhood  in  the  analogy).  Security  is  instituted  to  protect  this  system  from  the 
uncontrolled  part  of  the  environment. 

In  order  to  protect  a  system,  an  organization  will  rely  on  a  variety  of  products,  including 
devices  such  as  firewalls,  intrusion  detection  systems,  and  smart  card  readers  (locks  and 
sensors  in  the  analogy).  In  addition,  software  that  runs  on  system  nodes,  such  as  operat¬ 
ing  systems  that  identify  users  and  limit  access  to  files  and  browsers  that  provide  secure 
communication  to  services,  contribute.  The  security  of  the  system  depends  on  the  prod¬ 
ucts  that  are  used,  the  specific  security  algorithms  (such  as  cryptography)  (these  are 
mechanisms  in  locks  and  sensors  in  the  analogy),  the  way  that  they  are  incorporated  into 
the  system  architecture  (placement),  and  the  way  that  they  are  used.  The  system  devel¬ 
oper  does  not  solely  rely  on  tested  products,  but  also  tests  his  overall  system  through  cer¬ 
tification  and  accreditation.  C&A  is  designed  to  quantify  the  risks  inherent  in  the  system 
as  a  whole.  Since  the  products  are  tested  individually,  C&A  can  worry  less  about  the 
product  details  and  focus  more  on  their  arrangement,  inter-relationships  and  the  end  ef¬ 
fect. 
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Evaluation  of  information  systems  is  similar  to  that  described  for  the  house.  Each  system 
must  be  certified  and  accredited  that  its  overall  design  does  not  have  unacceptable  weak¬ 
nesses  and  that  the  various  security  products  have  been  properly  installed  as  spelled  out 
in  (the  DoD  Infonnation  Technology  Security  and  Certification  Process  (DITSCAP),  De¬ 
fense  Information  Assurance  Certification  and  Accreditation  Process  (DIACAP),  or  the 
National  Information  Assurance  Certification  and  Accreditation  Process  (NIACAP)).  In¬ 
dividual  security  products  are  tested  in  laboratories  to  verify  that  they  provide  the  stated 
security  claims  (Common  Criteria  Testing  Laboratories  (CCTL)).  Other  laboratories  are 
used  to  verily  that  a  product  properly  implements  some  algorithm  or  feature  (Federal  In¬ 
formation  Processing  (FIPS)  laboratories)).  Because  of  the  laboratory  analysis  done  un¬ 
der  FIPS,  the  NIAP  evaluation  does  not  need  to  examine  algorithm  or  feature  implemen¬ 
tation  (e.g.,  the  strength  of  encryption  between  network  nodes).  Because  of  the  evalua¬ 
tion  done  by  NIAP,  the  system  certification  does  not  need  to  do  detail  testing  of  product 
related  security  features  (e.g.,  the  strength  of  authentication  at  the  administrator’s  con¬ 
sole).  The  NIAP,  as  described  in  this  report,  focuses  on  the  area  that  is  between  algorithm 
certification  and  system  certification  and  accreditation.  While  the  NIAP  is  currently  re¬ 
stricted  to  IA  or  IA-enabled  products,  other  products  may  be  considered  as  potential 
sources  of  vulnerabilities. 

2.3.3  Product  Evaluation  Business  Case 

It  is  assumed  that  product  evaluation,  used  properly  is  a  positive  contributor  to  the  cyber¬ 
security  of  systems.  Properly  executed  product  evaluation  will  reduce  the  burden  of 
C&A  and  pay  for  itself  by  reuse  in  the  many  systems  that  use  the  product.  Without  prod¬ 
uct  evaluation,  the  properties  of  the  individual  products  are  examined  each  time  a  C&A  is 
done  on  a  system  that  uses  that  product.  These  evaluations  may  be  at  different  levels  of 
sophistication,  depending  upon  the  expertise  available  to  C&A.  Product  evaluation  itself 
becomes  more  valuable  when  it  provides  useful,  reusable  information  about  products  that 
can  be  integrated  with  C&A.  Product  evaluation  becomes  less  valuable  when  its  infor¬ 
mation  is  not  useful,  obscure,  unavailable,  or  not  able  to  be  integrated  with  system 
evaluations.  Different  users  have  differing  senses  of  value  depending  on  where  they  re¬ 
side  in  the  framework,  what  they  are  willing  to  risk,  and  what  they  can  afford  to  pay  for 
the  value  they  desire.  Various  users  and  consumers  of  the  NIAP  bring  differing  senses  of 
what  the  value-proposition  means.  At  the  system  layer,  a  product  evaluation  has  no 
meaning  unless  the  product  is  configured  and  used  in  the  same  way  as  it  was  evaluated. 
Also,  the  contribution  of  a  product  to  the  overall  security  value  of  a  system  depends  on 
the  placement  of  the  product  in  the  overall  system  architecture:  putting  a  vault  door  on  a 
house  adds  nothing  if  the  windows  are  left  open  and  unprotected.  This  inferred,  and  often 
unstated,  view  of  what  constitutes  value  creates  both  misunderstanding  and  confusion 
about  what  the  NIAP  is,  does,  and  is  expected  to  accomplish.  There  is  an  intrinsic  as¬ 
sumption  that  building  a  house  out  of  specified  and  tested  products  will  improve  the 
overall  worth  of  the  house.  The  same  is  assumed  for  building  a  security  system  out  of 
specified  and  evaluated  security  products  that  use  tested  and  evaluated  algorithms. 

2.3.4  Cybersecurity  Landscape  Summary 

This  framework  sets  the  stage  for  the  approach  and  methodology  the  review  team  under¬ 
took  and  the  subsequent  interpretation  of  the  NIAP  review  results.  The  framework  for 
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this  review  and  presentation  of  results  can  be  portrayed  in  two  dimensions  of  capability 
versus  cost.  The  objective  of  the  review  was  to  collect  quantitative  evidence  to  judge  the 
NIAP  efficacy  (value-proposition)  and  its  affordability  (cost-effectiveness);  as  might  be 
expected  the  further  up  the  “value-proposition-chain”  one  moves,  the  more  difficult  at¬ 
taining  quantifiable  evidence  of  the  security  worthiness  of  and  for  a  particular  instance  of 
a  particular  product  implementation  under  constrained  or  unknown  conditions  of  use  be¬ 
comes.  This  is  further  discussed  in  later  sections  of  the  document.  Additionally,  the 
complexity  of  presenting  options  makes  the  determination  of  precise  costs  beyond  the 
scope  of  this  analysis.  Rough  order  of  magnitude  data  will  be  provided  which  is  the  best 
estimate  (without  analysis)  of  the  individuals  involved  in  the  study.  Once  decisions  are 
made  on  which  options  to  pursue,  it  is  recommended  that  the  NIAP  be  asked  to  provide 
cost  estimation. 

2 . 4  Terminology 

Information  Assurance  (IA)  and  cybersecurity  are  terms  with  established  definitions. 
However,  the  terms  are  often  used  loosely  and  in  contexts  where  the  established  defini¬ 
tions  are  not  widely  understood  or  known;  and  the  body  of  knowledge,  skills,  and  tech¬ 
nologies  are  even  less  well  known  by  the  user.  Rapidly  changing  technology  and  a 
changing  threat  environment  are  largely  responsible  for  the  lack  of  stability  in  the  use  of 
tenninology.  Moreover,  the  meanings  of  terms  such  as  cybersecurity,  computer  security, 
network  security,  information  security,  and  information  assurance  change  depending  on 
who  is  using  them  and  in  what  context.  Annex  C  provides  a  glossary  of  terms.  At  the 
end  of  Annex  C  there  is  further  background  on  several  of  the  terms  used  in  this  report. 

2.4.1  Nomenclature 

The  complexity  of  the  subject  of  the  NIAP  in-context  of  the  cyberspace  landscape  and  the 
steps  taken  to  improve  overall  cybersecurity,  leads  to  the  need  for  a  labeling  convention 
that  is  used  throughout  the  report.  In  citing  Findings  we  will  use  a  tag  [Fx-n],  where  F  is 
for  finding,  x  is  one  or  more  characters  that  provide  reference  to  the  analysis  segment  that 
generated  the  finding,  and  n  is  the  number  of  findings  of  this  type  (for  example  FPCy-1 
in  Chapter  3  is  the  first  Finding  under  Policy  issues  for  Cybersecurity).  Similarly,  we 
have  labeled  Observations  [Ox-n],  Assumptions  [Ax-n],  Conjectures  [Cx-n],  Expecta¬ 
tions  [Ex-n],  and  Recommendations  [Rx-n],  These  tags  are  for  reference  to  derived  con¬ 
clusions  and  act  as  an  aid  for  traceability. 

2 . 5  Communities  of  Interest 

When  describing  the  evaluation  procedures  for  products  and  systems,  a  discussion  of  the 
various  communities  they  pertain  to  is  needed  as  different  communities  have  different 
requirements.  It  is  significant  that  the  NIAP  must  serve  all  of  these  communities  as  well 
as  the  broader  communities  outside  of  the  Federal  government,  with  a  product  certifica¬ 
tion  process  that  will  raise  the  basic  security  level  of  all  of  the  participants  in  and  out  of 
government.  The  Common  Criteria  itself  was  designed  for  the  broader  community  and 
the  U.S.  component  must  serve  that  broader  constituency.  However,  U.S.  Government 
stakeholders  often  have  differing  requirements  for  and  emphasis  on  product  evaluation. 
This  would  indicate  that  some  level  of  consolidation  for  at  least  process  and  evaluation 
methods  could  reduce  overall  complexity.  Figure  3  below  show  the  relationship  between 
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the  various  communities  addressed  by  C&A  standards:  Federal  Govermnent,  DoD,  Na¬ 
tional  Security,  and  Intelligence.  The  bottom  cylinder  consists  of  all  Departments  and 
Agencies  in  the  Executive,  Legislative,  and  Judicial  branches  of  the  Federal  Government. 
The  DoD,  as  one  of  the  Departments  of  the  Executive  Branch,  is  perched  above.  There  is 
considerable  overlap  between  the  DoD  and  the  national  security  and  intelligence  commu¬ 
nities  positioned  next  to  the  DoD  cylinder. 

The  national  security  community  of  the  Executive  Branch  is  an  amalgam  of  institutions: 

•  State  (including  our  embassies  and  consulates,  plus  our  overseas  communications 
and  foreign  assistance  programs), 

•  Defense  (encompassing  the  armed  forces  and  very  large  intelligence  components), 

•  Central  Intelligence  Agency  (CIA), 

•  Department  of  Homeland  Security  (DHS), 

•  Department  of  Justice  (including  the  FBI  and  the  Drug  Enforcement  Administra¬ 
tion)  and 

•  Often  also  the  Departments  of  Commerce,  Treasury,  and  Energy. 

At  the  top  of  the  diagram  is  the  intelligence  community  as  established  by  Executive  Or¬ 
der  12333,  United  States  Intelligence  Activities,  consisting  of  the 

•  CIA; 

•  National  Security  Agency  (NSA); 

•  Defense  Intelligence  Agency  (DIA); 

•  Offices  within  the  DoD  for  the  collection  of  specialized  national  foreign  intelli¬ 
gence  through  reconnaissance  programs; 

•  Bureau  of  Intelligence  and  Research  of  the  Department  of  State; 

•  Intelligence  elements  of  the  Army,  Navy,  Air  Force,  and  Marine  Corps,  the  Fed¬ 
eral  Bureau  of  Investigation  (FBI),  the  Department  of  the  Treasury,  and  the  De¬ 
partment  of  Energy;  and 

•  Staff  elements  of  the  Director  of  Central  Intelligence. 

As  these  descriptions  and  the  Figure  show,  there  is  considerable  overlap  and  some  separa¬ 
tion  among  the  communities.  Not  shown  is  the  commercial  industry,  parts  of  which  are 
considered  critical  infrastructure,  as  they  are  not  stakeholders  at  this  time,  but  do  contrib¬ 
ute  to  the  overall  cybersecurity  posture.  This  further  complicates  the  problem  of  policy 
and  guidance,  which  will  be  discussed  in  the  next  chapter. 
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2.5.1  A  Notional  Performance  Indicator/Cost  Trade  Space 

Ultimately  the  need  for  improvements  in  any  product  or  scheme  must  be  based  upon  the 
performance  cost  trade  space  shown  in  Figure  4.  The  figure  shows  the  trade  space  be¬ 
tween  cost  and  performance  and  the  transitions  that  can  be  undertaken  to  either  improve 
performance,  or  reduce  the  price.  There  are  three  such  transitions  shown  on  the  chart. 
The  first  shows  a  transition  to  a  lower  operating  performance  curve,  to  reduce  cost.  Here, 
a  decision  to  sacrifice  performance  (which  may  be  above  minimum  thresholds)  to  reduce 
costs  is  made.  A  second  transition  is  to  make  improvements  in  the  current  system  with 
the  trade  being  increased  cost  for  increase  performance.  The  third  transition  is  to  a 
higher-performance  operating  curve.  In  this  case,  a  new  approach  and  a  demand  for 
higher  performance  are  being  sought.  The  key  is  to  have  good  data  on  both  the  perform¬ 
ance  and  cost  of  the  current  operating  environment  so  that  estimates  can  be  made  for  each 
of  the  transitions.  It  is  the  goal  of  this  analysis  to  provide  trade  space  options.  However, 
insufficient  data  was  available  to  fully  exercise  this  method  of  assessment  to  the  degree 
necessary  to  make  a  fully  informed  business  case  for  the  outcome  effectiveness  of  prod¬ 
uct  evaluations  versus  alternatives  and  their  potential  costs.  However,  the  paradigm  be¬ 
low  provided  a  basis  for  seeking  such  data  from  stakeholders.  The  notional  perform¬ 
ance/cost  indicators  shown  below  are  input/output  indicators  to  NIAP.  The  review  sought 
but  did  not  discover  sufficient  and  detailed  data  to  apply  this  method  of  review  to  out¬ 
come  effectiveness  as  a  performance  indicator  nor  costs  of  potential  alternative  capabili¬ 
ties.  However,  the  notional  method  did  help  inform  and  structure  the  development  of  op¬ 
tions  for  a  way  ahead. 
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Figure  4.  Notional  Performance/Cost  Trade  Space 
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3 .  Policy  Review 


3 . 1  Scope  and  Context 

This  is  the  first  in  a  set  of  three  independent  analyses  of  the  Cybersecurity  landscape  and 
NIAP  as  it  fits  into  that  landscape.  Since  Federal  and  agency  policies  ultimately  dictate 
requirements,  IDA  explored  policy  to  determine  what  the  NIAP  must  be.  Policy  and  the 
strategies  for  implementing  policy  are  found  in  several  types  of  documents  including: 

•  Federal  statutes,  Executive  Orders  and  Presidential  Directives; 

•  Standards  or  department-level  directives; 

•  Administration  strategy  documents; 

•  Administration  and  congressional  reports. 

The  first  two  types  define  official  policy.  They  say  what  needs  to  be  done,  who  is  respon¬ 
sible  for  doing  it  and  give  some  expectation  of  how  it  is  to  be  done,  if  there  is  a  preferred 
method. 

Administration  strategy  documents  describe  an  administration’s  approach  to  implement¬ 
ing  policy.  Administration  and  congressional  reports  provide  insight  into  the  difficulties 
facing  Federal  departments  by  documenting  progress  (or  lack  thereof)  in  implementing 
policy. 

Annex  A  lists  the  wide  range  of  policy  documents  that  we  reviewed  for  their  relevance  to 
the  NIAP  process.  Annex  D  contains  detailed  discussions  of  the  relevant  specifics  of  each 
document. 

3 . 2  Themes 

The  discussion  of  requirements  is  organized  around  five  themes  that  occur  across  poli¬ 
cies: 

1 .  Cybersecurity; 

2.  Standards; 

3.  Research; 

4.  Education,  Training  &  Awareness  (ET&A);  and 

5.  Acquisition. 

Some  policies  may  contain  requirements  in  each  area;  other  policies  focus  specifically  on 
a  single  theme.  The  relationships  among  the  various  policies  are  generally  organized 
thematically  and  hierarchically.  Analysis  of  these  relationships  provides  a  more  under¬ 
standable  picture  of  the  policy  landscape.  A  section  summarizing  Findings  and  Recom¬ 
mendations  for  each  theme  concludes  this  chapter. 
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3.2.1  Cybersecurity  Policies  and  NIAP 

In  this  report,  the  tenns  cybersecurity  and  information  assurance  are  used  interchangea¬ 
bly  as  their  definitions  are  quite  similar.  The  term  Cybersecurity  gained  formal  status 
when  The  Department  of  Homeland  Security  Authorization  Act  for  Fiscal  Year  2005 
[DHS2005]  amended  the  Paperwork  Reduction  Act  to  define  it  as: 

the  prevention  of  damage  to,  the  protection  of,  and  the  restoration  of  computers, 
electronic  communications  systems,  electronic  communication  services,  wire 
communications,  and  electronic  communications,  including  information  con¬ 
tained  therein,  to  ensure  its  availability,  integrity,  authentication,  confidentiality, 
and  nonrepudiation. 

The  definition  given  in  the  National  Information  Systems  Security  (INFOSEC)  Glossary 
[NST2000c]  for  information  assurance  is 

Conducting  those  operations  that  protect  and  defend  information  and  information 
systems  by  ensuring  availability,  integrity,  authentication,  confidentiality,  and 
non-repudiation.  This  includes  providing  for  restoration  of  information  systems 
by  incoiporating  protection,  detection,  and  reaction  capabilities. 

Both  cybersecurity  and  information  assurance  encompass  the  “five  pillars”  of  informa¬ 
tion  assurance — availability,  integrity,  authentication,  confidentiality,  and  nonrepudiation 
of  information  systems  as  well  as  the  concepts  of  protection  and  restoration.  Cybersecu¬ 
rity  refers  explicitly  to  computers  and  electronic  systems,  whereas  information  assurance 
refers  more  broadly  to  information  systems,  which  might  or  might  not  be  electronic. 

The  NIAP  is  both  an  organization  and  a  process  created  to  address  a  specific  requirement 
for  cybersecurity  for  a  specific  community  of  interest.  To  the  extent  that  it  is  successful,  it 
contributes  directly  to  the  overall  increase  in  security  of  National  Security  Systems 
(NSSs),  and  indirectly  to  Federal  government  and  private  sector  products. 

Chapter  One  described  how  the  NIAP  was  created  by  agreement  between  NSA  and  NIST 
to  assist  both  organizations  in  fulfilling  their  statutory  cybersecurity  responsibilities  under 
PL  100-235  (Computer  Security  Act  of  1987).  Annex  E  provides  a  more  complete  history 
of  NIAP.  Initial  funding  and  staffing  were  provided  per  agreement  by  both  organizations, 
with  the  understanding  that  the  majority  of  future  funding  should  come  from  commercial 
testing  laboratories  conducting  Common  Criteria-based  evaluations  of  IT  products  on  a 
fee-for-service  basis. 

The  expectation  that  the  majority  of  the  NIAP  funding  would  be  provided  by  the  fees 
charged  to  companies  undergoing  the  certification  and  validation  process  has  failed  to 
materialize.  For  one  thing,  government  oversight  is  mandated  in  evaluations.  This  over¬ 
sight  is  not  paid  by  the  evaluation  itself  and  increasing  amounts  of  resources  have  been 
diverted  to  oversight  of  a  growing  evaluation  program.  Second,  the  National  Voluntary 
Lab  Accreditation  Program  (NVLAP  -  this  program  certifies  CCEVS  commercial  labora¬ 
tories)  has  the  right  to  charge  laboratories  fees  associated  with  their  evaluation  and  certi¬ 
fication  under  CCEVS.  However,  the  fees  barely  cover  the  costs  accrued  by  NVLAP. 
Finally,  the  labs  doing  the  certification  do  not  provide  funds  for  the  other  activities  the 
NIAP  was  expected  to  support.  At  the  same  time,  new  statutes  and  priorities  from  both 
parent  organizations  have  changed  the  mix  of  activities  that  can  be  supported,  leaving 
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little  discretionary  funding  for  activities  like  the  NIAP.  The  result  has  been  that  the  NIAP 
has  shed  functions  that  were  not  directly  related  to  evaluation. 

The  Computer  Security  Division  (CSD)  is  the  division  within  NIST  responsible  for  carry¬ 
ing  out  NIST’s  mandate  in  cybersecurity.  FISMA  requires  an  annual  report  from  NIST  on 
the  status  of  its  activities  required  by  the  statute.  In  the  2003  report  (the  first  one  after 
FISMA  went  into  effect),  the  CSD  reported  the  current  status  of  its  activities  and  a  state¬ 
ment  that  “...along  with  many  other  NIST  units,  [CSD]  is  taking  a  significant  budget  cut 
in  2004.  The  work  planned  for  2004,  as  described  in  this  report...  is  very  conditional. 
This  budget  cut  will  delay  and  curtail  some  of  the  planned  work...” 3  4  The  cuts  men¬ 
tioned  limited  NIST’s  participation  in  the  NIAP  to  less  than  1  staff  year  and  the  NVLAP 
-  the  program  that  certifies  commercial  laboratories  to  do  product  evaluations.  The  de¬ 
crease  in  budget  for  NIST  is  symptomatic  of  the  shift  in  priority  away  from  supporting 
the  NIAP. 

The  Information  and  Security  Privacy  Advisory  Board  (ISPAB)  completed  a  report  on 
funding  for  the  cybersecurity  program  at  NIST  and  provided  their  findings  to  OMB.  The 
details  of  the  funding  show  an  inconsistent  level  of  funding,  with  some  years  showing  a 
significant  growth,  while  others  a  significant  decrease,  unrelated  to  tasking  from  Con¬ 
gress  or  OMB.  As  the  report  states,  the  most  recent  “unfunded  mandates”  put  a  severe 
strain  on  NIST’s  ability  to  meet  its  commitments  and  expectations,  including  support  for 
the  NIAP.  5  Since  the  existence  of  the  NIAP  is  not  a  Congressional  or  Federal  require¬ 
ment,  the  priority  for  support  has  fallen  below  other  requirements,  resulting  in  NIST’s 
almost  complete  withdrawal  from  participation  in  the  NIAP.  The  ISPAB  strongly  recom¬ 
mended  that  the  NIAP  be  properly  funded  to  address  activities  required  for  the  federal 
civilian  sector  and  to  ensure  balance  in  the  NIAP’s  activities. 

In  testimony  before  the  House  Committee  on  Government  Subcommittee  on  Technology, 
Information  Policy,  Intergovernmental  Relations  and  the  Census  on  16  March  2004,  Dep¬ 
uty  Undersecretary  of  Commerce  for  Technology,  Secretary  Benjamin  H.  Wu,  laid  out  the 
activities  NIST  has  accomplished  since  the  enactment  of  FISMA,  emphasizing  the  priori¬ 
ties  and  funding  challenges  facing  NIST.  The  Commerce  Department  had  requested  a 
funding  increase  for  FY  2005  to  address  these  priorities,  including  that  required  for 
NIST’s  participation  in  the  NIAP.6 

OMB  acknowledged  the  importance  of  the  work  that  NIST  does  and  reported  on  a  num¬ 
ber  of  NIST’s  activities  in  both  the  FY03  and  FY04  FISMA  Reports  to  Congress.7  OMB 


3  FISMA,  Sec  303,  para  (d)(10). 

4  NIST  CSD,  2003  Annual  Report  “Welcome”  by  Edward  Roback,  Division  Chief.  Pg  2. 

5  ISPAB  report,  “A  Report  by  the  Information  Security  and  Privacy  Advisory  Board”,  June  2004,  Docu¬ 
ment  is  available  at: 

http://csrc.nist.gov/ispab/bd-recommendations/ISPAB-ReportAdequateFundingNIST-CSD.pdf 

6  Wu,  Ben.  “Statement  Before  the  Committee  on  Government  Reform  Subcommittee  on  Technology,  In¬ 
formation  Policy,  Intergovernmental  Relations  and  the  Census,  U.S.  House  of  Representatives”,  16  March 
2004.  Document  is  available  at: 

http://www.technology.gov/Testimony/BHW_040316.htm. 

7  OMB,  “FY2003  Report  to  Congress  on  Federal  Government  Information  Security  Management”,  1  March 
2004.  Document  is  available  at:  http://www.whitehouse.gov/omb/inforeg/fy03_fisma_report.pdf. 
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included  mention  of  NIST’s  participation  in  the  NIAP,  but  did  not  comment  on  either  the 
priority  or  sufficiency  of  either  that  participation  or  any  of  NIST’s  activities. 

The  Cyber  Security  Industry  Alliance,  a  relatively  new  industry  group  has  taken  on  a 
number  of  issues  in  cybersecurity,  one  of  which  is  the  NIAP.  It  issued  a  report  in  July 
2004  with  a  number  of  recommendations  to  improve  the  NIAP  process,  which  will  be 
discussed  in  a  later  chapter.* * * 8  Later  in  2004,  CSIA  issued  its  “Agenda  for  the  Next  Ad¬ 
ministration”  and  made  specific  recommendations  concerning  NIST  funding  and 
strengthening  of  the  NIAP  certification9. 

NSA,  whose  budget  goes  through  DoD,  has  historically  been  successful  in  obtaining 
funding  for  its  various  programs.  However,  the  support  provided  to  the  NIAP  is  not  iden¬ 
tified  in  a  specific  line  item  that  can  be  identified  and  monitored  to  determine  its  suffi¬ 
ciency.  None-the-less,  funds  have  been  provided  to  the  NIAP  for  the  execution  of  it  pro¬ 
gram  since  its  inception.  Further,  NSA  has  funded  the  development  of  over  70  protection 
profiles  for  use  in  evaluation  of  products  for  defense  requirements.  The  continuing  need 
for  product  evaluation  in  the  defense  sector  is  only  a  small  portion  of  the  NIAP  need. 

The  principal  documents  concerning  the  NIAP  for  DoD  are  embodied  in  NSTISSP-11 
and  DoD  INST  8500.1.  NSTISSP-11  requires  the  use  of  evaluated  products  for  defense 
and  national  security  systems.  DoD  INST  8500.1  requires  that  product  claims  be  based 
upon  a  protection  profile,  if  one  exists.  Both  of  these  are  essential  elements  in  the  cyber¬ 
security  process.  The  principle  document  for  Certification  and  Accreditation  (C&A)  is 
the  DITSCAP,  which  as  of  this  date  is  not  integrated  with  the  NIAP  product  evaluations. 

Annex  D  discusses  the  cybersecurity  statues,  EOP  documents,  and  Federal  Agency  poli¬ 
cies  in  detail.  DHS,  by  virtue  of  its  broader  role,  will  be  monitoring  applications  that  are 
required  to  meet  in  totality,  virtually  all  of  the  documents  mentioned  in  Annex  D. 

3.2.2  Standards  and  Guidelines 

For  cybersecurity,  the  establishment  of  standards  is  critical  to  the  ability  of  an  organiza¬ 
tion  to  evaluate,  acquire  and  manage  applications,  products  or  services.  Congress  recog¬ 
nized  this  need  and  designated  responsibilities  for  development  of  standards  and  guide¬ 
lines  for  cybersecurity  for  the  Federal  government.  In  addition  to  standards  and  guide¬ 
lines  are  best  practices  and  protocols. 

For  Federal  agencies,  the  complex  mix  of  mandatory,  voluntary  standards  and  best  prac¬ 
tices  (see  Table  1)  pose  severe  challenges  in  implementation  for  any  security  official. 
Federal  agencies  may  have  to  deal  with  three  different,  and  potentially  inconsistent,  sets 


“FY2004  Report  to  Congress  on  Federal  Government  Information  Security  Management”,  1  March  2005. 

Document  is  available  at : 

http://www.whitehouse.gov/omb/inforeg/2004_fisma_report.pdf. 

8  CSIA,  ‘‘NIAP  Certification:  Proposals  by  CSIA  for  Strengthening  Security  Certification”,  23  July  2004. 
Organization  web  site  at  https://www.csialliance.org/home. 

9  CSIA,  “Agenda  for  the  Next  Administration:  Proposals  by  the  Cyber  Security  Industry  Alliance”,  7  De¬ 
cember  2004.  Document  is  available  at : 

https://www.csialliance.org/resources/pdfs/Agenda_for_Next_Administration.pdf 


20 


of  standards.  Heads  of  these  agencies  have  the  authority  to  adopt  more  stringent  stan¬ 
dards  for  their  entire  organization,  which  could  mean  adoption  of  the  most  stringent  of 
the  three  sets  of  standards.  Most  choose  not  to  for  budgetary  reasons.  In  the  case  of  the 
NIAP  and  the  use  of  evaluated  products,  only  DoD  and  the  NS  S  must  currently  comply. 
More  detail  on  NIST,  NSS,  DoD  and  IC  standards  and  guidelines  can  be  found  in  Annex 
D. 


Table  1.  Mandatory/Voluntary  Standards  Matrix 


Standards  for  Fed¬ 
eral  Systems  re: 
cybersecurity 

NIST 

NSA  (for  na¬ 
tional  security 
systems) 

DoD 

Intelligence 

community 

ISO/IEEE/ANSI/ASTM 

International  Gov¬ 
ernmental  organi¬ 
zations 

Regulatory  Bod¬ 
ies  (GAO,  FTC, 
FCC,  SEC,  FDA, 
NRC  etc) 

Mandatory 

Federal  Gov¬ 
ernment  (incl 

contractors  & 

grantees) 

FIPS 

NSTISSC/NTSS 

P 

DoD  Issuances 

DSCID  6/3, 
supplemented  by 
specific  IC  issu¬ 
ances 

Deference  in  Lieu  of  Developing  a  Man¬ 
datory  Standard:  An  agency  may  decide 
that  it  does  not  need  to  issue  a  mandatory 
regulation  because  voluntary  compliance 
with  either  an  existing  standard  or  one  de¬ 
veloped  for  the  purpose  will  suffice  for 
meeting  the  needs  of  the  agency 

Code  of  Federal 

Regulations 

National  (in¬ 
cludes  S/L/T 

&  private 
sector) 

n/a 

applies  to  NG 

tbd  (IRA  2004) 

Code  of  Federal 

Regulations 

International 

(government 
&  private 
sector) 

CCEVS 

selective  appli¬ 
cability  to 
coalition  opera¬ 
tions 

n/a 

NATO 

collaboration  w/US 

entities  may  re¬ 
quire  compliance 

w/CFR 

Voluntary 

Federal  Gov¬ 
ernment 

Special 

Pubs,  Fed¬ 
eral  Agency 
Security 

Practices 

(BP) 

security  configu¬ 
ration  guides 

STIGs  (Secu¬ 
rity  technical 
implementation 
guides) 

n/a 

n/a 

National  (in¬ 
cludes  S/L/T 

&  private 
sector) 

Special  Pubs 

security  configu¬ 
ration  guides 

STIGs  (Secu¬ 
rity  technical 
implementation 
guides) 

n/a 

Best  practices 
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Standards  for  Fed¬ 
eral  Systems  re: 
cybersecurity 

NIST 

NSA  (for  na¬ 
tional  security 

systems) 

DoD 

Intelligence 

community 

ISO/IEEE/ANSI/ASTM 

International  Gov¬ 
ernmental  organi¬ 
zations 

Regulatory  Bod¬ 
ies  (GAO,  FTC, 
FCC,  SEC,  FDA, 
NRC  etc) 

International 

(government 
&  private 
sector) 

Special  Pubs 

security  configu¬ 
ration  guides 

STIGs  (Secu¬ 
rity  technical 
implementation 
guides) 

n/a 

ISO  17799 

collaboration  wAJS 

entities  may  en¬ 
courage  adherence 
to  US  best  prac¬ 
tices 

3.2.3  Research  Policy  and  NIAP 

As  cybersecurity  technology  is  still  relatively  immature,  research  is  critical  to  ensuring 
that  NIAP-relevant  cybersecurity  solutions  are  developed  at  the  same  pace  as  changes  in 
IT  technology.  The  Federal  government  is  dependent  upon  the  private  sector  for  the  ma¬ 
jority  of  cybersecurity  research,  but  Congress  and  the  EOP  recognize  that  Federal  agen¬ 
cies  need  to  play  an  active  role.  The  NIAP  is  both  a  recipient  of  the  benefits  of  this  re¬ 
search  activity  by  the  private  sector  and  academia,  and  also  an  active  participant,  by  de¬ 
sign. 

Policy  makers  wrongly  assume  that  adequate  research  to  support  the  NIAP  needs  now 
and  in  the  future  will  be  conducted. 10  While  a  number  of  reports  and  documents  discuss 
the  need  for  cybersecurity,  most  are  silent  on  the  subject  of  research  relevant  to  product 
evaluation  and  security  evaluation  metrics.  In  addition,  any  cross-pollination  of  research 
results  is  achieved  on  an  ad  hoc  basis;  no  mechanisms  exist  to  enable  the  identification 
and  transition  of  product  evaluation  and  security  evaluation  metrics  research  results  into 
the  NIAP  community.  Detailed  discussion  of  the  relevant  documents  can  be  found  at  An¬ 
nex  D. 


In  testimony  to  the  US  House  of  Representatives  Committee  on  Science  in  May  2003,  DARPA  Director  Dr.  Tony  Tether  described 
DARPA's  past  investments  in  information  assurance  and  cybersecurity  research.  This  work  included  development  and  improvement 
of  firewalls,  intrusion  detection  methods,  and  intrusion  tolerance  techniques  that  allow  systems  to  operate  through  attacks.  The  bulk  of 
this  research,  from  DARPA's  point  of  view,  has  been  completed  and  DARPA  has  moved  on  to  more  advanced  concepts  of  cognitive 
computing  in  which  computer  systems  are  expected  to  know  what  they  are  doing  —  including  knowing  when  they  are  under  attack  and 
how  to  respond.  DARPA's  original  firewall  research  has  matured  into  widely  used  commercial  products.  The  remainder  of  this  re¬ 
search  is  still  in  the  proof  of  concept  and  product  development  pipeline. 

Also  in  2003,  NSF  started  a  process  to  reinvigorate  their  Cyber  Trust  program.  In  2004,  NSF  funded  50  research  projects  addressing 
different  aspects  of  computer  and  network  security,  privacy,  and  trust.  Virtually  all  of  this  research  focuses  on  fundamental  research 
questions  that  have  long-range  implications.  This  work,  however,  is  not  intended  to  address  today's  immediate  computer  security 
problems.  Research  results  that  can  be  transitioned  quickly  into  products  and  applications  are  good,  but  this  is  not  a  criteria  NSF  uses 
in  selecting  research  projects  for  funding. 
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3.2.4  Education,  Training  &  Awareness  (ET&A)  Policy  and  NIAP 

ET&A  is  a  critical  component  of  any  information  security  program,  long  recognized  by 
Congress,  OMB  and  the  Federal  IT  community.  As  with  research,  the  NIAP  is  both  a 
beneficiary  of  ET&A  programs  and  a  potential  contributor  to  the  increase  in  the  body  of 
knowledge  regarding  cybersecurity.  A  knowledgeable  and  competent  user  and  practitio¬ 
ner  community  is  necessary  for  a  successful  cybersecurity  program.  Requirements  and 
specific  assignments  of  responsibility  for  this  activity  are  contained  in  numerous  policy 
documents  described  in  Annex  D. 

The  NIST  documents  (NIST  SP  800-16  and  NIST  SP  800-50)  lay  out  a  comprehensive 
ET&A  program,  which,  if  followed  consistently  by  the  Federal  Departments  and  agencies 
regardless  of  community  of  interest,  would  result  in  significant  improvements  in  work¬ 
force  awareness  and  competence  in  cybersecurity.  As  with  standards,  the  issue  is  more  in 
Federal  agency  consistent  implementation,  monitoring  and  enforcement.  This  is  a  re¬ 
quired  item  for  reporting  under  FISMA  2002.  OMB’s  FY2003  FISMA  Report  to  Con¬ 
gress  listed  insufficient  information  security  awareness  and  training  as  one  of  the  material 
weaknesses  in  reports  from  23  agencies.11  Representative  Davis’s  FY2004  FISMA 

Scorecard  reiterated  that  “...specialized  training  for  employees  with  significant  security 

12 

responsibilities”  remains  a  challenge  for  Federal  agencies. 

That  said,  there  are  two  additional  concerns  to  consistent  implementation  regarding 
ET&A.  One  of  the  concerns  is  that  the  detailed  training  requirements  were  issued  in 
1998,  concurrent  with  the  establishment  of  the  NIAP,  therefore,  any  benefit  to  the  NIAP 
process  from  ET&A  and  vice  versa  was  not  available.  Additionally,  the  requirements 
have  not  kept  pace  with  technology  and  the  change  in  management  of  IT  infrastructures. 
An  update  to  the  ET&A  implementation  documents  issued  by  NIST,  taking  into  account 
that  both  the  experience  from  the  NIAP  and  a  reflection  of  maturity  in  Federal  IT  man¬ 
agement,  would  be  extremely  valuable.  The  second  concern  is  that  the  detail  provided  for 
functions  and  training  requirements  for  both  evaluators  and  certificate  users  is  insufficient 
regarding  product  certification  and  how  that  is  useful  in  system  certification  and  accredi¬ 
tation. 


3.2.5  Acquisition  Policy  and  NIAP 

Acquisition  policy  is  the  area  that  is  most  clearly  tied  to  the  NIAP.  Cybersecurity  is  one 
of  a  number  of  factors  that  must  be  considered  in  Federal  Department  and  Agency  IT 

n 

capital  planning  for  their  enterprise.  This  approach,  with  significant  oversight  from 
OMB  through  the  budgetary  process,  focuses  on  performance  and  outcomes  from  a  sys- 


11  OMB,  “FY  2003  Report  to  Congress  on  Federal  Government  Information  Security  Management”,,  1 
March  2004,  p.  23.  document  available  at  http://www.whitehouse.gov/omb/inforeg/fy03_fisma_report.pdf. 

12  Chairman,  Flouse  Government  Reform  Committee,  Representative  Tom  Davis,  “Statement  on  2004  Fed¬ 
eral  Computer  Security  Report  Card  Grades”,  16  February  2005.  document  available  at 
http://reform.house.gov/GovReform/News/DocumentSingle.aspx?DocumentID=6813t 

13  NIST  provides  guidance  to  Federal  agencies  through  the  following  document:  NIST  SP  800-65,  “Inte¬ 
grating  IT  Security  into  the  Capital  Planning  and  Investment  Control  Process”,  January  2005.  Document 
available  at  http://csrc.nist.gov/publications/nistpubs/800-65/SP-800-65-Final.pdf. 
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terns  perspective.  To  the  extent  that  a  specific  IT  security  product  performs  as  specified 
and  produces  desired  outcomes  within  the  enterprise  infrastructure,  a  particular  Federal 
department  or  agency  will  make  a  buy  decision.  If  that  product  has  been  evaluated,  so 
much  the  better,  but  that  is  not  the  primary  factor,  and  for  these  agencies,  there  is  no  re¬ 
quirement  to  do  so.  NIST  does  provide  guidance  to  Federal  agencies  on  how  to  choose 
evaluated  products. 14 

For  NSS  and  DoD  who  are  required  to  use  evaluated  products,  the  primary  factor  is 
whether  the  product  has  been  evaluated,  not  necessarily  that  it  performs  the  desired  func¬ 
tions  and  produces  the  desired  outcomes  in  the  best  way  possible  for  the  enterprise.  It 
may  be  that  the  best  available  product  has  not  been  evaluated  and  the  manufacturer  has 
no  intention  of  doing  so.  In  that  case,  NSS  and  DoD  must  rely  on  a  lesser  product  whose 
manufacturer  has  been  willing  to  go  through  the  NIAP  process.  There  are  also  situations 
where  no  product  in  a  particular  category  has  been  evaluated. 

3 . 3  Summary  of  Policy  Findings  &  Recommendations 

Reviewing  the  policy  landscape  and  how  policy  around  and  about  the  NIAP  has  been  im¬ 
plemented  has  identified  a  number  of  issues  that  are  listed  in  this  section  as  Findings. 
Some  of  the  issues  involve  the  NIAP  itself,  most  are  external  to  the  NIAP  and  involve 
agencies  across  the  Federal  government.  The  Recommendations  describe  a  desired  activ¬ 
ity  to  resolve  the  issues  described  in  the  Finding.  Where  possible,  a  specific  agency  with 
responsibility  in  an  area  of  concern  is  identified  in  the  Recommendation. 

3.3.1  Cybersecurity 

The  question  is  whether  the  NIAP  process  should  be  made  mandatory  for  more  than  the 
NSS  and  DoD.  There  is  also  the  concern  about  how  to  integrate  an  evaluated  product, 
whose  configuration  when  evaluated  may  change,  into  the  certification  and  accreditation 
process  for  government  IT  systems. 

Finding  [FPCy-1] 

The  complex  policy  landscape  and  lack  of  a  single  source  for  current  and  superseded 
policies  makes  it  difficult  for  Federal  Departments  and  Agencies  to  determine  the  re¬ 
quirements  for  their  particular  situation. 

Recommendation  [RPCy-1] 

There  needs  to  be  a  single  source,  available  to  all,  that  maintains  copies  of  all  current  and 
superseded  policy  documents.  This  source  should  be  available  on-line,  with  sophisticated 
search  capability.  A  single  Federal  organization  (such  as  NIST  or  DHS/NSCD)  should  be 
responsible  for  this  activity  and  be  resourced  accordingly. 


14  NIST  SP  800-36,  “Guide  to  Selecting  Information  Technology  Security  Products”,  October  2003.  Docu¬ 
ment  available  at  http://csrc.nist.gov/publications/nistpubs/800-36/NIST-SP800-36.pdf.  and  NIST  SP  800- 
23,  “Guidelines  to  Federal  Organizations  on  Security  Assurance  and  Acquisition/Use  of  Tested/Evaluated 
Products”,  August  2000.  Document  available  at  http://csrc.nist.gov/publications/nistpubs/800-23/sp800- 
23.pdf 
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Finding  [FPCy-2] 

The  NIAP  was  created  by  agreement  between  two  agencies  in  different  Federal  depart¬ 
ments,  with  no  formal  recognition  by  OMB  or  Congress.  It  has  no  official  standing  other 
than  the  agreement,  which  can  be  modified,  rescinded,  or  terminated  unilaterally  by  ei¬ 
ther  of  the  two  parent  agencies. 

Recommendation  [RPCy-2] 

If  it  is  determined  that  the  NIAP  provides  a  valuable  service,  it  should  be  formally  recog¬ 
nized  and  chartered  by  the  appropriate  entity.  This  chartering  would  include  specific  de¬ 
scriptions  of  the  responsibilities  of  the  two  parent  organizations  regarding  resources 
(funding,  staffing,  and  facilities)  and  management. 

Finding  [FPCy-3] 

The  NIAP’s  budget  is  not  a  line  item  in  either  parent  agency’s  budget,  preventing  detailed 
oversight  of  the  budget  process  to  determine  sufficiency  and  justification.  This  has  re¬ 
sulted  in  decreasing  available  resources  as  the  parent  agencies  address  more  pressing  is¬ 
sues. 


Recommendation  [RPCy-3] 

The  budget  for  the  NIAP  should  be  identified  by  specific  line  item  and  justified  in  the 
parent  agency’s  (DoD/NSA  and  Department  of  Commerce/NIST)  budget.  This  will  allow 
for  oversight  to  ensure  sufficient  funding  for  the  NIAP’s  mission. 

3.3.2  Standards 

Finding  [FPSt-1] 

The  National  Technology  Transfer  and  Advancement  Act  [NTTAA]  1995  requires  the  use 
of  voluntary  consensus  standards  in  lieu  of  developing  Federal  standards.  For  Federal 
agencies  to  detennine  which  standards  to  use,  requires  the  mapping  of  existing  voluntary 
consensus  standards,  Federal  standards  (FIPS)  and  standards  from  the  different  communi¬ 
ties  of  interest  to  detennine  gaps,  conflicts  and  overlaps.  To  this  date,  this  mapping  has 
not  been  done,  or  if  it  has,  was  not  evident  to  the  researchers  in  this  study. 

Finding  [FPSt-2] 

The  existence  of  the  communities  of  interest  results  in  differing  sets  of  potentially  con¬ 
flicting  and  non-interoperable  requirements.  While  Federal  Information  Processing  Stan¬ 
dards  (FIPS)  issued  by  NIST  are  mandatory  for  the  Federal  government,  National  secu¬ 
rity  systems  (NSS)  are  exempt  from  them.  NSS  have  their  own  standards  and  guidelines 
as  does  the  DoD  and  Intelligence  Community.  Annex  D  provides  more  detail  on  the 
standards  for  these  communities  of  interest. 

Although  Federal  Departments  or  Agencies  are  at  liberty  to  establish  a  single  set  of  re¬ 
quirements  for  their  activity  based  on  the  most  stringent  requirements,  it  may  not  be  fea¬ 
sible  for  all  to  do  so.  CSA  1987,  CCA  1996  and  FISMA  2002  encourage  NIST  to  work 
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with  DoD  and  NS  A  to  synchronize  standards,  the  ability  to  achieve  much  in  this  area  de¬ 
pends  upon  emphasis  and  resources. 

Recommendation  [RPSt-1] 

NIST  should  work  with  NSA,  DoD  and  the  IC  to  synchronize  existing  and  future  stan¬ 
dards  to  reduce  the  potential  of  conflicting  and  non-interoperable  requirements  among  the 
communities.  While  NIST  development  is  open  and  input  is  solicited  from  all  entities,  a 
more  proactive  formation  of  consortiums  and  alliances  is  suggested. 

3.3.3  Research 

Finding  [FPRe-1] 

Current  levels  of  cybersecurity  research  funding  are  inadequate  and  fail  to  address  current 
cybersecurity  issues,  much  less  those  of  the  future  (such  as  grid  computing,  distributed 
intelligent  agent  systems,  distributed  knowledge  management,  composable  systems,  and 
systems  of  systems.)  (See  Footnote  10). 

Finding  [FPRe-2] 

A  timeline,  schedule,  and  process  for  addressing  future  cybersecurity  concerns  are  lack¬ 
ing. 


Finding  [FPRe-3] 

A  process  for  coordinating  government  efforts  and  allocating  research  resources  for  cy¬ 
bersecurity  research  is  lacking. 

Recommendations  [RPRe-1] 

DoD  and  NIST  develop  a  strategic  plan  for  investigation  of  future  NIAP -related  cyberse¬ 
curity  needs  and  conduct  the  research  required  to  address  the  needs.  OSTP  should  set 
national  cybersecurity  research  priorities  and  the  research  agenda  and  coordinate  research 
efforts  among  government  organizations.  OSTP  should  develop  a  mechanism  for  coordi¬ 
nation  of  government  cybersecurity  research  efforts. 

Finding  [FPRe-4] 

The  memorandum  establishing  the  NIAP  (see  Annex  G)  says  that  NIAP  seeks  to  foster 
research  and  development  in  security  tests,  methods,  and  metrics  but  to  date  this  has  not 
happened. 

Finding  [FPRe-5] 

The  memorandum  establishing  the  NIAP  (see  Annex  G)  says  that  NIAP  seeks  to  foster 
research  and  development  in  security  tests,  methods,  and  metrics,  however,  there  is  no 
plan  or  process  in  place  to  identify  the  necessary  research  or  to  insure  that  the  research  is 
performed. 
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Recommendation  [RPRe-2] 

NIAP  should  develop  a  plan  and  undertake  the  research  required  to  foster  research  and 
development  in  security  tests,  methods,  and  metrics. 

Finding  [FPRe-6] 

While  some  portion  of  information  security  research  advances  achieved  as  a  result  of  the 
Homeland  Security  Act  of  2002  may  be  of  use  for  the  NIAP,  no  process  is  in  place  to 
identify  those  results  or  to  transfer  them  into  the  CCEVS  process. 

Finding  [FPRe-7] 

The  need  for  research  to  improve  the  quality  of  our  cyber  defense  and  defensive  informa¬ 
tion  operations  is  widely  acknowledged  in  numerous  government  documents,  however, 
no  document  points  to  the  NIAP  as  a  tool  for  addressing  this  need  or  directs  research  that 
would  result  in  the  NIAP  being  able  to  address  this  need.  In  the  few  cases  where  the  need 
for  research  is  identified  in  a  document,  it  is  usually  cited  in  relation  to  a  specific  threat, 
such  as  mobile  malicious  code,  and  never  addressed  toward  improving  the  capability  of 
determining  if  software  is  secure  via  the  NIAP  process15. 

Finding  [FPRe-8] 

While  the  Clinger-Cohen  Act  imposes  a  research  duty  upon  the  National  Science  Founda¬ 
tion  (NSF),  there  is  no  mechanism  for  communicating  NIAP’s  needs  to  the  NSF  or  for 
extracting  NSF  research  advances  and  employing  them  for  NIAP  purposes. 

Recommendation  [RPRe-3] 

NIAP  should  develop,  under  OSTP  guidance,  a  process  to  identify  and  transition  success¬ 
ful  NIAP-related  research  to  the  NIAP  laboratories,  developers,  and  evaluators.  OSTP 
should  develop  a  process  to  insure  successful  transition  of  cybersecurity  research  results 
across  government  organizations. 

3.3,4  Education,  Training  &  Awareness 

Finding  [FPEta-1] 

The  training  documentation  produced  by  NIST  for  the  Federal  Government  (SP800-16) 
in  1998  was  issued  concurrently  with  the  establishment  of  the  NIAP.  Although  a  model 
program  when  it  was  issued,  the  changes  in  policy,  technology,  management  of  Federal 
IT  organizations,  and  organization,  require  a  similar  significant  update  to  this  document. 

Finding  [FPEta-2] 

The  level  of  detail  in  the  current  NIST  SP800-16  is  insufficient  to  ensure  an  appropriate 
level  of  knowledge  for  either  the  NIAP  certificate  users  or  certifiers  (see  also  5.3.1,  5.3.2, 


15  For  the  purposes  of  this  report,  research  into  proof  of  correctness  of  software  is  not  considered  to  be 
equivalent  to  research  into  proof  of  security  capabilities  and  quality  of  software. 
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and  5.3.4.)  This  detail  is  necessary  for  performance  and  evaluation  of  perfonnance  for 
those  functions. 

Recommendation  [RPEta-1] 

The  NIST  ET&A  documents  (not  all  of  which  have  been  mentioned)  need  to  be  updated 
to  address  changes  in  policy,  technology,  management  and  organization  of  Federal  IT. 
Additionally,  sufficient  detail  should  be  included  to  address  the  deficiencies  noted  for  the 
NIAP  certificate  users  and  certifiers  (see  also  5.3.1,  5.3.2,  and  5.3.4.) 

3.3.5  Acquisition 

Finding  [FPAq-1] 

While  FISMA  documents  specify  the  security  controls  that  must  be  implemented  in  Fed¬ 
eral  unclassified  systems,  there  is  no  requirement  on  how  Federal  agencies  must  choose 
those  products.  The  requirement  for  acquisition  of  evaluated  IA/IA -related  products  only 
exists  for  the  NSS  and  DoD.  Outside  of  this  no  acquisition  policy  concerning  IA/IA- 
related  products  exists  for  Federal  departments/agencies  since  the  Federal  Infonnation 
Resources  Management  Regulations  (FIRMR)  was  rescinded  in  1996.  This  means  that 
the  rest  of  the  Federal  Government  can  choose  security  products  based  on  their  own  crite¬ 
ria  that  may  or  may  not  have  been  evaluated. 

Recommendation  [RPAq-1] 

Consideration  should  be  given  to  make  an  improved  NIAP  product  evaluation  process16 
mandatory  for  all  Federal  Government  entities.  This  must  be  conditioned  upon  several 
actions  that  improve  the  usefulness  of  the  product  and,  where  possible,  reduces  overall 
costs.  The  resources  necessary  to  accomplish  this  must  also  be  available. 

Recommendation  [RPAq-2] 

An  official  statement  of  policy  from  the  appropriate  entity  should  be  made  that  the  NIAP 
is  the  U.S.  Common  Criteria  Evaluation  and  Validation  Scheme  to  achieve  the  require¬ 
ment  for  DoD  to  use  evaluated  products.  Under  mutual  recognition,  the  DoD  could  use 
products  evaluated  under  other  national  schemes  as  well.  This  should  be  done  only  after 
improvements  are  made  under  options  4  and  5  of  Chapter  8. 

Finding  [FPAq-2] 

The  NIAP  process  leading  to  product  certification  does  not  directly  contribute  to  systems 
certification  required  by  statute  and  OMB. 

Recommendation  [RPAq-3] 

NIST  and  NSA  should  make  the  relationship  explicit  between  the  NIAP  process  and  the 
process  for  certification  and  accreditation  of  systems.  This  includes  descriptions  of  how 


16  Improved  as  outlined  in  either  option  4  or  5  of  Chapter  8.  These  improvements  should  result  in  effective 
and  integrated  product  evaluations. 
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to  use  the  products  from  the  NIAP  process  to  feed  into  the  C&A  process.  This  is  a  re¬ 
quirement  for  the  recommendations  above. 
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4. 


NIAP  and  Evolution 


This  chapter  describes  the  NIAP  and  how  it  has  evolved  from  its  inception  to  its  current 
organization,  responsibilities,  and  operations.  This  is  the  second  in  a  set  of  three  inde¬ 
pendent  analyses  of  the  Cybersecurity  landscape  and  NIAP  as  it  fits  into  that  landscape. 
The  timeline  of  activities  that  led  up  to  the  formation  of  the  NIAP  and  the  flurry  of  activi¬ 
ties  that  followed  are  included  in  Annex  E.  The  timeline  at  the  end  of  Annex  E  shows 
that  the  development  of  this  partnership  and  the  approach  to  product  evaluation  was  not  a 
spur  of  the  moment  undertaking.  The  NIAP  is  an  outgrowth  of  the  many  initiatives  in 
cybersecurity  by  both  NIST  and  NSA  and  in  response  to  the  policy  landscape  described 
in  the  previous  chapter.  It  was  well  thought  out  by  a  number  of  professionals  and  in¬ 
tended  to  provide  a  viable  product  evaluation  scheme  as  a  part  of  an  overall  cybersecurity 
program. 

4 , 1  The  NIAP’s  Original  Charter 

As  described  in  Section  1,  the  NIAP  is  a  joint  effort  between  the  NIST  and  the  NSA  to 
provide  technical  leadership  in  the  research  and  development  of  security-related  informa¬ 
tion  technology  test  methods  and  assurance  techniques.  The  Terms  of  Reference  docu¬ 
ment  that  established  their  collaboration  [NIST/NSA1998]  set  forth  the  following  goals 
for  the  NIAP: 

•  Promote  the  development  and  use  of  evaluated  IT  products  and  systems; 

•  Champion  the  development  and  use  of  national  and  international  standards  for  IT 
security; 

•  Foster  research  and  development  in  IT  security  requirements  definition,  test 
methods,  tools,  techniques,  and  assurance  metrics; 

•  Support  a  framework  for  international  recognition  and  acceptance  of  IT  security 
testing  and  evaluation  results;  and 

•  Facilitate  the  development  and  growth  of  a  commercial  security  testing  industry 
within  the  U.S. 

The  most  significant  component  of  NIST  and  NSA’s  collaboration  is  the  Common  Crite¬ 
ria  Evaluation  and  Validation  Scheme  (CCEVS).  CCEVS  is  the  NIAP’s  product  evalua¬ 
tion  process,  which  has  evolved  from  NSA’s  earlier  Trusted  Product  Evaluation  Program 
(TPEP).  As  the  “CC”  in  its  name  implies,  CCEVS  is  based  on  the  International  Common 
Criteria  for  Information  Technology  Security  Evaluation  [ISO  International  Standard 
15408].  CCEVS  supports  many  of  the  goals  outlined  above. 
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NIST  and  NSA  separately  fund  (or  do  not  fund)  their  respective  contributions  to  the 
NIAP.  NIST  contributions  have  been  minimal.  The  two  agencies  have  separate  respon¬ 
sibilities  for  IT  security  under  the  Computer  Security  Act  of  1987  and  later  legislation. 
The  NIAP  activities,  therefore,  reflect  each  agency’s  priorities.  Both  agencies  agreed  that 
evaluations  were  the  highest  priority  and  the  result  is  CCEVS.  Beyond  this,  NIST  con¬ 
ducts  laboratory  accreditations  and  has  developed  training  materials,  primarily  for  pro¬ 
spective  evaluators.  NSA  has  developed  evaluation  training,  and  a  number  of  Protection 
Profiles  (PP’s),  which  provide  standard  evaluation  criteria  for  products  of  a  particular 
type,  such  as  firewalls.  These  PP’s  are  intended  primarily  for  DoD  use.  NSA  has  played 
an  important  role  in  interpreting  the  letter  and  intent  of  the  Common  Criteria  when  ques¬ 
tions  have  been  raised  during  evaluations.  They  have  also  been  heavily  involved  in  the 
development  and  evolution  of  the  CC. 

Areas  identified  in  NIAP’s  goals  that  have  not  received  the  same  level  of  attention  as  the 
CCEVS  include  IT  security  research,  development  of  tools  for  security  testing,  and  met¬ 
rics  for  assurance. 

4 . 2  CCEVS  for  IT  Security 

A  formally  established  group  called  the  CCEVS  Validation  Body  is  responsible  for  the 
operation  of  the  validation  scheme.  The  Validation  Body’s  principal  objective  is  to  en¬ 
sure  the  provision  of  IT  security  evaluation  and  validation  services  for  both  government 
and  industry17.  This  group  is  formally  established  in  response  to  the  Common  Criteria 
Recognition  Agreement  (CCRA). 

The  NIAP  established  the  following  objectives  in  developing,  operating,  and  maintaining 
the  evaluation  and  validation  scheme  : 

To  meet  the  needs  of  government  and  industry  for  cost-effective  evaluation  of  IT 
products; 

To  encourage  the  formation  of  commercial  security  testing  laboratories  and  the 
development  of  a  private  sector  security  testing  industry; 

To  ensure  that  security  evaluations  of  IT  products  are  performed  to  consistent 
standards;  and 

To  improve  the  availability  of  evaluated  IT  products. 

The  validation  scheme  is  intended  to  serve  many  communities  of  interest  with  very  di¬ 
verse  roles  and  responsibilities.  This  community  includes  IT  product  developers,  product 
vendors,  value-added  resellers,  systems  integrators,  IT  security  researchers,  acquisition/ 
procurement  authorities,  IT  product  consumers,  auditors,  and  accreditors  (individuals  de¬ 
ciding  the  fitness  for  operation  of  those  products  within  their  respective  organizations). 
Close  cooperation  between  government  and  industry  is  paramount  to  the  success  of  the 
scheme  and  the  realization  of  its  objectives. 


17  http://niap.nist.gov/cc-scheme/ 

18  http://niap.nist.gov/cc-scheme/ccevs-objectives.html 
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Commercial  testing  laboratories  called  Common  Criteria  Testing  Laboratories  (CCTL) 
carry  out  the  NIAP  product  evaluations.  These  laboratories  are  accredited  by  the  NIST’s 
National  Voluntary  Laboratory  Accreditation  Program  (NVLAP)  and  approved  by  the 
CCEVS  Validation  Body.  NVLAP  accreditation  is  one  of  the  requirements  for  becoming 
a  CCTL.  The  purpose  of  NVLAP  accreditation  is  to  ensure  that  laboratories  meet  the  re¬ 
quirements  of  ISO/IEC  17025:2005,  General  Requirement  for  the  Competence  of  Cali¬ 
bration  and  Testing  Laboratories  [IS02005]  and  the  specific  scheme  requirements  for  IT 
security  evaluation. 

During  the  course  of  an  evaluation,  the  CCEVS  Validation  Body  provides  technical  guid¬ 
ance  to  those  testing  laboratories,  validates  the  results  of  IT  security  evaluations  for  con¬ 
formance  to  the  CC,  and  serves  as  an  interface  to  other  nations  for  the  recognition  of  such 
evaluations. 

Upon  completion  of  a  successful  evaluation,  the  CCEVS  Validation  Body  issues  the 
evaluated  product  or  protection  profile  a  CC  Certificate.  This  certificate  confirms  that  the 
evaluation  was  conducted  by  an  accredited  laboratory  using  the  Common  Evaluation 
Methodology  (CEM),  and  that  the  conclusions  of  the  testing  laboratory  are  consistent 
with  the  evidence  presented  during  the  evaluation. 

All  IT  products  and  protection  profiles  that  have  successfully  completed  evaluation  and 
validation  appear  on  the  Common  Criteria  Validated  Products  List  (VPL).  This  list  in¬ 
cludes  those  products  and  profiles  successfully  completing  similar  processes  under  the 
schemes  of  authorized  signatories  to  the  Arrangement  on  the  Mutual  Recognition  of 
Common  Criteria  Certificates  in  the  Field  of  Information  Technology  Security.  The  US 
CCEVS  has  currently  been  limited  to  products  that  claim  some  information  assurance 
(IA)  properties.  This  is  mandated  in  NTSSISP-11  for  defense  critical  and  intelligence  IT 
system  applications,  and  is  true  of  those  on  the  lists.  This  has  not  been  applied  to  the 
wide  range  of  products  that  may  reside  in  an  IT  system,  and  affect  its  security. 

4.2.1  Evaluation  Process 

The  NIAP  product  evaluations  follow  a  well-established  process  that  involves  at  least 
four  sets  of  participants.  The  players  are: 

1 .  The  sponsor,  who  is  often  the  product  developer  or  their  agent; 

2.  An  evaluation  laboratory; 

3.  The  CCEVS  Validation  Body;  and 

4.  An  evaluation  validator  assigned  by  the  CCEVS  Validation  Body. 

The  steps  in  the  process  are  shown  in  Figure  5.  The  starting  point  is  when  the  sponsor 
determines  the  need  for  an  evaluation.  The  sponsor  must  decide  if  the  product  is  to  be 
evaluated  against  an  existing  protection  profile  (PP19)  or  against  a  unique  set  of  claims. 


19  The  protection  profile  is  a  set  of  requirements  both  functional  and  assurance  for  a  class  of  products  (such 
as  a  firewall).  It  may  specify  precisely,  or  allow  certain  latitude  in  achieving  these  functionalities.  It  is 
normally  developed  by  a  set  of  domain  experts  in  a  community  of  interest.  NSA  has  developed  a  number 
of  PPs  for  DoD  use. 
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These  claims  have  to  be  documented  in  what  the  Common  Criteria  (CC)  calls  a  security 
target  (ST).  An  ST  is  required  for  every  evaluation.  Claims  of  conformance  to  PPs  are 
not  partial  and  any  claim  means  that  all  of  the  provisions  of  the  PP  are  covered  in  the  ST. 
If  any  provision  of  the  PP  is  removed  from  the  ST,  the  claim  to  confonnance  to  the  PP 
must  be  dropped.  PPs  have  predetermined  evaluation  assurance  levels  (EALs).  If  a  PP  is 
not  used,  the  EAL  (or  a  set  of  assurance  requirements)  for  the  evaluation  must  also  be 
decided.  While  the  CC  makes  no  requirement  on  the  choice  of  assurance  requirements, 
an  EAL  package  is  generally  selected  (exceptions  do  exist). 
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Figure  5.  Overview  of  the  NIAP  Evaluation  Process 
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Figure  5  shows  the  process  for  evaluations  at  EAL4  and  below.  Evaluations  above  EAL- 
4  follow  a  similar  process  with  a  fifth  participant,  NSA,  taking  responsibility  for  addi¬ 
tional  testing  and  vulnerability  analyses.  The  additional  testing  and  analysis  is  done  after 
the  lab  has  completed  their  testing  and  submitted  their  Evaluation  Test  Report  (ETR). 

The  next  step  in  the  process  is  for  the  sponsor  to  contract  with  a  lab  to  conduct  the 
evaluation.  The  lab  then  puts  together  an  evaluation  proposal  package  consisting  of  the 
sponsor's  product  description  and  security  target,  and  their  work  plan  and  schedule.  This 
proposal  package  is  then  submitted  to  the  CCEVS  Validation  Body  for  approval.  If  the 
CCEVS  finds  discrepancies  in  the  proposal,  they  return  comments  to  the  lab  and  sponsor 
for  resolution. 

Once  the  CCEVS  Validation  Body  accepts  the  proposed  evaluation  they  assign  a  valida¬ 
tor  to  monitor  the  evaluation  as  it  progresses.  The  validator  works  with  the  lab  through¬ 
out  the  evaluation  to  ensure  that  it  is  conducted  as  planned  and  meets  all  CC  require¬ 
ments.  When  the  validator  is  satisfied  with  all  the  details  in  the  lab's  work  plan,  they  hold 
a  kickoff  meeting  to  formally  start  the  evaluation.  At  this  time,  the  product  is  entered  on 
CCEVS’  official  list  of  products  in  evaluation. 

In  conducting  the  evaluation,  the  lab  typically  starts  several  parallel  activities  to  review 
product  documentation  and  prepare  for  testing.  For  example,  the  security  target  claims 
are  reviewed  and  a  test  plan  is  developed.  The  test  plan  covers  the  requirements  of  the 
EAL  as  delineated  in  the  Common  Evaluation  Methodology  (CEM).  Below  EAL4,  test¬ 
ing  is  minimal  and  survey  in  nature.  Other  documentation  includes  the  product's  configu¬ 
ration  management  record,  description  of  the  product's  design,  user  manuals,  and  product 
delivery  procedures.  Documentation  may  include  the  low-level  and  high-level  design 
documents,  correspondence  documents,  and  security  interface  specifications  and  sum¬ 
mary  specifications.  No  specific  format  is  required  by  the  standard,  only  content.  How¬ 
ever,  in  practice  separately  prepared  documents  are  generated  in  lieu  of  usage  of  the  de¬ 
velopmental  documentation.  Preparation  of  these  documents  and  the  preparation  and  re¬ 
vision  of  the  ST  may  be  undertaken  by  the  developer,  or  by  a  consultant  hired  by  the  de¬ 
veloper.  In  many  instances,  the  developer  will  contract  with  a  separate  group  at  the 
evaluation  lab  for  the  preparation  of  evidence,  which  is  allowed  under  CCEVS  rules,  as 
long  as  separation  between  evaluation  and  evidence  preparation  is  demonstrated.  Any 
discrepancies  the  lab  finds  are  brought  to  the  sponsor  and  developers'  attention  for  resolu¬ 
tion. 

The  validator  monitors  the  lab's  activities  throughout  the  evaluation.  This  monitoring 
includes  review  of  ETR  sections  for  compliance  with  CC  and  CCEVS,  witnessing  tests, 
and  other  oversight  activities.  At  the  conclusion  of  the  evaluation  process,  the  validator 
must  accept  the  lab’s  ETR.  To  ensure  the  evaluation  remains  on  track,  several  milestones 
are  typically  observed.  For  example,  the  lab  will  have  the  validator  approve  their  test 
plan  before  proceeding  to  conduct  the  tests.  During  testing,  the  lab  will  have  the  validator 
check  their  procedures  and  validate  the  results  they  are  collecting.  Keeping  the  validator 
in  the  loop  minimizes  the  possibilities  of  surprises  later  in  the  process. 

If  testing  reveals  discrepancies;  that  is,  if  the  product  fails  to  meet  a  security  target  claim, 
the  sponsor  and  developer  are  engaged  to  resolve  the  problem.  If  the  product's  problems 
cannot  be  corrected,  the  claims  it  is  unable  to  meet  must  be  removed  from  the  security 
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target.  Any  changes  to  the  ST  are  first  reviewed  and  commented  on  by  the  validator.  The 
validator  can  also  raise  significant  ST  issues  with  the  CCEVS  Validation  Body  to  keep 
them  aware  of  lowered  product  expectations.  The  CCEVS  Validation  Body  might  want  to 
know,  for  example,  that  a  product  that  was  being  evaluated  against  a  PP  is  not  able  to 
meet  all  of  its  requirements.  This  may  result  in  dropping  a  claim  to  PP  perfonnance  or 
changes  to  the  software,  hardware  or  firmware  as  needed  to  meet  PP  conformance.  In  the 
latter  case,  delays  in  the  evaluation  may  occur. 

Vulnerability  analysis  is  usually  the  final  stage  of  the  evaluation  process.  Checking  for 
vulnerabilities  begins  at  EAL2,  but  serious  vulnerability  testing  (AVA_VLA.3)  is  ordi¬ 
narily  not  required  until  above  EAL4.  As  with  other  testing,  any  vulnerability  discrepan¬ 
cies  that  are  found  are  raised  with  the  sponsor  and  developer  for  resolution.  Occasion¬ 
ally,  analysis  may  discover  very  obscure  vulnerabilities  that  have  little  chance  of  being 
exploited.  If  the  developer  is  unable  to  correct  the  problem,  the  product  may  still  com¬ 
plete  and  pass  its  evaluation  by  modifying  the  ST  as  to  threat  or  objectives.  The  valida¬ 
tor,  in  this  case,  may  question  that  the  vulnerability  is  applicable  for  systems  built  from 
components  at  this  evaluation’s  assurance  level.  This  ruling  may  be  appealed  to  CCEVS. 

Alternatively,  the  threat  environment  in  the  ST  may  be  modified  so  that  no  claims  to 
counter  that  particular  vulnerability  are  made.  If  a  PP  is  claimed  for  conformance,  the  ST 
cannot  lower  the  threat  expectation  and  still  claim  PP  conformance.  DoD  requires  the  ST 
be  based  upon  an  NSA  approved  PP.  At  the  conclusion  of  documentation  reviews,  the 
lab  will  finalize  its  Evaluation  Technical  Report  and  submit  it,  along  with  the  ETR,  to  the 
CCEVS  Director.  When  the  Director,  in  conjunction  with  CCEVS  Chief  Validation 
Body,  accepts  the  findings  of  the  evaluation  and  the  validation  report,  a  certificate  is  is¬ 
sued  for  the  product  and  the  product  is  placed  on  the  Validated  Products  List.  Two  lists 
were  mentioned,  and  both  are  significant  to  DoD.  The  first  is  the  list  of  products  in 
evaluation  and  the  second  is  the  VPL.  NSTISSP-11  requires  a  product  to  be  on  one  of 
these  two  lists  for  use  in  defense  or  national  security  systems. 

Sources  used  in  this  analysis  included  the  Common  Criteria  [CC2004a,b,c],  CCEVS 
evaluation  process  descriptions  [CCEVS  11 999],  [CCEVS22000],  [CCEVS32002], 
CCEVS42001],  and  [CCEVS52000],  and  informal  interviews  with  NIAP  personnel. 

4.2.2  Built-In  Assumptions  and  Associated  Risks 

Our  analysis  of  the  current  NIAP  evaluation  scheme  uncovered  a  number  of  assumptions 
that  appear  to  have  driven  the  formulation  of  these  processes.  Many  of  these  assumptions 
were  recognized  in  developing  the  CCEVS,  and  checks  and  balances  were  included  in  the 
scheme  to  ensure  that  potential  problems  would  be  contained.  Other  assumptions  appear 
to  be  implicit,  as  evidenced  by  how  evaluations  are  conducted  under  the  CCEVS.  These 
assumptions  have  fewer  safeguards  and  represent  larger  risks.  Both  sets  of  assumptions 
were  derived  by  CCEVS  documentation,  observation  and  personal  experience  of  analysts 
on  the  evaluation  team. 

4.2.2. 1  Evaluated  Products  Assumptions 

The  first  set  of  assumptions  concerns  the  products  that  have  been  through  evaluations  and 
how  and  where  they  are  used. 
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Component  Security  [AN-01] 

Evaluated  components  are  assumed  to  provide  better  security  for  the  systems  of  which 
they  are  a  part.  It  is  further  assumed  that  system  integrators  will  build  systems  intelli¬ 
gently  using  appropriate  components.  (These  are  key  assumptions.  If  there  are  excep¬ 
tions,  evaluated  products  may  not  live  up  to  expectations.) 

Dubious  Evaluations  [AN-02] 

The  CC  allows  evaluations  of  products  that  provide  little  or  no  protection.  Examples  in¬ 
clude  partial  product  evaluations,  trivial  security  requirements,  and  evaluations  of  simple 
electrical  components.  The  CCEVS  provides  no  safeguards  against  such  evaluation 
abuses.  Certificates  must  be  issued  for  products  that  pass  these  evaluations,  even  though 
they  provide  little  if  any  useful  protection.  However,  Validation  Reports  should  articulate 
shortfalls  and  differences  between  the  portions  of  the  product  that  were  evaluated  and  the 
product  as  used  in  a  typical  configuration.  It  should  be  noted  that  the  user  decides  use¬ 
fulness,  and  the  concern  becomes  significantly  less  when  Education  Training  and  Aware¬ 
ness  have  provided  most  users  with  the  information  to  read  and  interpret  an  ST.  Such 
education  would  allow  the  user  to  better  evaluate  the  ST  claims. 

4.2.2.2  Evaluation  Processes  and  Results  Assumptions 

The  next  group  of  assumptions  concerns  the  methods  and  techniques  used  in  evaluations, 
and  results  from  evaluations. 

Comparability  and  Consistency  [AN-03] 

Evaluations  based  on  the  Common  Criteria  are  assumed  to  be  comparable  and  consistent. 
Validation  is  tasked  with  providing  that  level  of  consistency.  That  is,  if  different  labs 
were  to  evaluate  the  same  product,  they  would  produce  virtually  identical  results. 
Evaluations  performed  in  other  countries  under  Mutual  Recognition  Agreements  (MRA’s) 
are  assumed  to  be  conducted  with  the  same  quality  and  consistency  as  under  the  NIAP’s 
CCEVS. 

Single  Evaluation  [AN-04] 

The  CC  assumes  a  security  evaluation  needs  to  be  perfonned  only  once,  worldwide,  for  a 
given  version  of  a  product  in  a  given  type  of  environment.  The  risk  here  is  that  informa¬ 
tion  systems  and  environments  are  constantly  changing,  often  without  full  consideration 
of  security  impacts.  The  Maintenance  Assurance  packages  do  make  provisions  for  con¬ 
tinued  and  partial  re-evaluations,  but  these  are  not  part  of  any  assurance  level  packages. 
Several  DoD  protection  profiles  do  invoke  these  assurance  requirements. 

Point  Evaluations  [AN-05] 

Each  evaluation  will  consider  only  one  specific  version  of  a  product,  for  a  specific  envi¬ 
ronment.  Even  if  there  is  no  assurance  maintenance  or  flaw  remediation  process  for  the 
product,  the  results  of  evaluations  are  assumed  to  be  useful  to  consumers  for  the  foresee¬ 
able  future.  This  last  point  is  a  conjecture. 
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Few  Protection  Profiles  [AN-06] 

PPs  were  intended  to  provide  standard  sets  of  claims  against  which  products  that  perform 
similar  security  functions  would  be  evaluated.  It  was  assumed  that  PPs  would  be  devel¬ 
oped  covering  a  wide  range  of  security  needs  and  that  these  PPs  would  serve  as  standards 
for  product  comparisons.  NS  A  has  created  some  70  PPs  to  address  military  security 
needs,  but  very  few  have  been  developed  or  adopted  for  commercial  and  civilian  gov¬ 
ernment  use.  These  PPs  have  not  been  extensively  vetted  through  consumer  and  private 
organizations,  are  generally  not  thought  of  as  available  for  general  use.  Few  product 
evaluations  (other  than  DoD  applications)  make  PP  claims,  leaving  consumers  with  little 
basis  for  comparing  competing  products.  Unless  a  product  is  meant  to  satisfy  an  identi¬ 
fied  military  need,  the  product  vendor  is  left  to  decide  the  security  claims  the  product  is 
evaluated  against.  While  the  development  of  PPs  to  cover  all  contingencies  should  not  be 
part  of  the  NIAP’s  requirements,  the  fostering  and  support  of  industry  consortiums  to  do 
this  should  be. 

Vulnerability  Testing  [AN-07] 

CC  evaluations  are  assumed  to  include  sufficient  analysis  and  testing  to  detect  obvious 
security  vulnerabilities.  The  CC,  however,  defines  only  a  general  process  for  describing 
attack  (e.g.,  level  of  effort  and  knowledge  required).  The  CEM  also  provides  only  gen¬ 
eral  guidance  on  various  types  of  attacks  to  consider. 

The  NIAP  requires  no  specific  vulnerability  testing  for  even  commonly  evaluated  prod¬ 
ucts  (such  as  firewalls,  Portable  Operating  System  Interface  (POSIX)  operating  systems, 
or  IDS  systems).  No  standard  test  suite  has  been  developed  for  vulnerability  testing. 
There  is  no  requirement  to  use  automated  vulnerability  analysis  tools  (such  as  source 
code  scanning  tools  or  input  injection  tools)  to  perform  vulnerability  analyses. 

Source  Code  Review  [AN-08] 

Evaluators  are  assumed  to  have  access  to  all  product  information  necessary  to  assess  the 
product’s  security  properties.  The  CC  does  not  require  evaluators  to  perform  independent 
reviews  of  product  source  code  for  security  purposes.  For  software  products  the  source 
code  is  ground  truth,  containing  essentially  everything  evaluators  need  to  know  about  its 
behavior.  Source  code,  however,  is  not  available  for  EAL  3  and  below.  Only  at  EAL4 
and  higher  is  a  complete  set  of  source  code  made  available.  Manual  reviews  of  source 
code  can  be  expensive  or  impossible  for  large  software  systems.  Automated  tools  can  be 
used  to  identify  critical  code  components  and  focus  reviews.  While  use  of  such  tools  is 
pennitted,  it  is  not  required  at  any  EAL.  This  is  both  a  NIAP  responsibility  and  a  CC  re¬ 
sponsibility. 

Coding  Errors  [AN-09] 

The  evaluation  process  assumes  that  developers  generally  know  how  to  implement  secure 
software;  that  is,  coding  errors  that  result  in  security  problems  are  rare  and  unlikely.  Oth¬ 
erwise,  code-checking  tools  would  be  applied  to  the  software.  Evaluations  concentrate  on 
the  requirements  and  design  documentation  instead  of  the  implementation.  Most  security 
vulnerabilities,  however,  can  be  traced  to  common  implementation  errors.  Unfortunately, 
relatively  few  developers  know  how  to  avoid  security  flaws  because  this  is  not  taught  at 
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most  universities.  Annex  F  includes  references  justifying  the  claim  that  most  vulnerabili¬ 
ties  are  caused  by  common  implementation  errors.  The  annex  also  shows  that  tools  can 
detect  many  of  these  vulnerabilities,  but  as  noted  above,  such  tools  are  not  required  by 
the  CC. 


Requirements  Tracing  [AN-10] 

Documentation  that  traces  requirements  through  design  and  implementation  (depending 
on  the  EAL)  is  assumed  to  be  an  effective  security  analysis  technique.  These  so-called 
representation  correspondence  requirements,  especially  at  the  upper  EAL,  impose  trace- 
ability  requirements  far  in  excess  of  typical  commercial  development  practice.  As  a  re¬ 
sult,  the  developer  must  spend  significant  effort  writing  or  re-writing  documentation;  in 
many  cases  long  after  product  development  has  been  completed.  This  documentation  ef¬ 
fort  rarely  contributes  to  finding  security  flaws. 

Documentation  Requirements  [AN-11] 

CC  product  documented  information  requirements  are  fixed  and  highly  inflexible.  They 
assume  that  all  development  methodologies  produce  the  same  detailed  documented  in¬ 
formation  and  that  resulting  products  are  susceptible  to  the  same  security  flaws.  Com¬ 
mercial  software  development  practices,  however,  differ  widely  and  rarely  produce  the 
classical  “waterfall  model”  documentation  expected  by  the  CC.  While  attention  to  well- 
established  software  development  and  documentation  practices  generally  corresponds 
with  better  product  quality  and  product  documentation,  product  security  requires  a  par¬ 
ticular  focus  that  is  not  assured  by  the  CC’s  documentation  requirements.  The  current 
requirements  lead  CC  evaluations  closer  to  evaluating  the  quality  of  documentation  in¬ 
stead  of  the  quality  of  the  product. 

Documentation  After  the  Fact  [AN-12] 

Product  design  documentation  provided  as  evidence  in  evaluations  is  assumed  to  repre¬ 
sent  the  design  the  product  was  actually  based  upon.  Developers,  however,  can  provide 
evaluators  design  documentation  prepared  after  the  product  has  been  completed,  which 
does  not  necessarily  reflect  how  the  product  was  developed.  A  third  party  who  may  or 
may  not  understand  how  the  software  was  developed  may  actually  produce  this  documen¬ 
tation.  In  general,  documentation  produced  late  would  be  acceptable  if  it  was  accurate. 

Evidence  Availability  [AN-13] 

Public  evaluation  results  are  assumed  to  provide  sufficient  information  about  test  results 
and  evidence  collected  in  evaluations  for  consumers  to  make  infonned  decisions  about 
products.  Details  of  test  results  and  other  evaluation  evidence  contained  in  Evaluation 
Technical  Reports,  however,  are  not  included  in  certificates  and  are  not  available  for  pub¬ 
lic  review.  There  is  a  need  to  make  these  reports  more  easily  understandable.  Proprietary 
interests  in  the  ETR  and  test  result  documents  needed  to  be  resolved. 

For  information  about  tools  to  support  evaluations  (e.g.,  for  proactive  measures,  vulner¬ 
ability  testing,  source  code  reviews),  see  Annex  F. 
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4.2.23  Assumptions  About  Product  Users 

These  assumptions  concern  IT  system  users,  customers,  consumers,  and  owners. 

Use  of  Evaluated  Products  [AN-14] 

To  comply  with  national  and  agency  information  assurance  policies,  information  system 
owners  must  buy  products  that  have  been  evaluated,  and  they  may  assume  this  means  that 
evaluated  products  meet  their  (or  someone’s  interpretation  of  their)  security  needs.  This 
may  be  partially  true  with  conformance  to  a  properly  vetted  PP.  Chapter  5  reviews  the 
expectations  in  this  area.  While  no  one  has  provided  this  assurance,  it  is  none-the-less  a 
problem. 

Diverse  Security  Requirements  and  Assurance  Needs  [AN-15] 

Not  all  infonnation  systems  have  the  same  security  policies  and  requirements.  Not  all 
information  system  owners  need  the  same  level  of  assurance  in  the  security  of  their  sys¬ 
tems.  Evaluated  products  are  assumed  to  support  construction  of  systems  with  different 
security  requirements  and  different  levels  of  assurance.  The  ultimate  responsibility  for 
use  of  the  product  is  with  the  system  developer.  He  should  be  given  broad  access  to 
evaluation  documentation. 

Paying  for  Evaluations  [AN-16] 

Security  is  assumed  to  be  important  enough  to  information  system  owners  (including  the 
US  government)  that  either  they  will  pay  enough  extra  for  evaluated  products  to  make 
evaluations  economically  viable  for  the  vendor,  or  they  will  pay  for  evaluations  them¬ 
selves. 


Smart  Consumers  [AN-17] 

IT  system  users  and  administrators  are  assumed  to  always  read  and  follow  all  security- 
related  guidance  (including  installation  guidance)  provided  to  them;  this  is  a  dubious  as¬ 
sumption.  Careful  attention  should  be  paid  to  default  configurations.  Consumers  are  as¬ 
sumed  to  know  their  security  requirements  and  purchase  only  those  evaluated  products 
that  meet  their  needs.  This  further  assumes  that  consumers  will  use  an  evaluated  product 
only  in  the  environment  in  which  it  was  evaluated,  or  that  the  Security  Target  and  public 
evaluation  material  provide  enough  information  for  them  to  understand  how  the  product 
will  perform  in  their  environment. 

4.2.2.4  Assumptions  About  Product  Developers 

These  assumptions  concern  the  behavior  of  product  developers. 

Full  Disclosure  [AN-18] 

Developers  are  assumed  to  disclose  all  information  relevant  to  product  evaluations,  such 
as  design  documentation  and  results  from  their  own  testing,  and  will  not  hide  knowledge 
of  vulnerabilities,  back  doors,  or  other  flaws  from  evaluators.  Because  of  the  distributed 
nature  of  many  software  development  projects,  the  developer  may  not  even  know  these 
things  himself.  However,  this  assumption  is  less  of  a  problem  if  the  education  training 
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and  awareness  of  developers  is  at  a  high  enough  level  among  developers  that  they  can 
precisely  articulate  what  they  know  and  don’t  know  about  their  products. 

Trustworthy  Developers  [AN-19] 

Product  developers  are  assumed  to  be  trustworthy  and  would  not  knowingly  insert  mali¬ 
cious  code  into  their  products.  Because  of  the  distributed  nature  of  many  software  devel¬ 
opment  projects,  the  developer  may  not  even  know  these  things  himself. 

Impact  of  Evaluation  Costs  [AN-20] 

The  cost  of  evaluations  is  assumed  to  be  insignificant  (or  at  least  not  prohibitive)  so  that 
small  businesses  and  independent  developers  can  have  products  evaluated  as  well  as  large 
corporations.  The  typical  cost  of  $100,000  or  more  for  an  evaluation  (with  additional 
costs  to  the  internal  processes  of  the  vendor),  however,  is  a  barrier  to  entry  for  open 
source  software  (OSS)  and  many  products  developed  by  small  businesses. 

4.2.2.5  Assumptions  About  Evaluation  Laboratories 

The  final  set  of  assumptions  concerns  the  evaluation  labs. 

Evaluation  Independence  [AN-21] 

A  product’s  security  evaluation  is  assumed  to  be  an  unbiased  assessment,  not  influenced 
by  the  developer  or  vendors  with  financial  interests  in  the  product.  This  is  also  a  require¬ 
ment  of  Handbook  150-20. 

Laboratory  Competence  [AN-22] 

Labs  that  are  approved  to  perform  evaluations  are  assumed  to  have  demonstrated  their 
competence  in  conducting  evaluations  and  maintain  that  competence.  In  addition  to  the 
laboratory’s  general  level  of  competence,  each  evaluation  team  within  that  lab  is  assumed 
to  be  competent.  This  is  also  a  requirement  of  Handbook  150-20. 

Commercial  Viability  [AN-23] 

The  demand  for  evaluations  is  assumed  to  be  high  enough  to  make  commercial  product 
evaluation  labs  viable. 

Laboratory  Competition  [AN-24] 

Competition  among  evaluation  labs  is  assumed  to  be  sufficient  to  keep  evaluation  costs  to 
a  minimum,  without  losing  quality.  This  is  also  a  requirement  of  Handbook  150-20. 

Flexibility  [AN-25] 

Evaluation  labs  are  assumed  to  be  flexible  in  their  ability  to  accommodate  shifts  in  the 
number  of  products,  new  and  different  types  of  products,  and  changes  in  security  tech¬ 
nology. 
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Proprietary  Evaluation  Processes  [AN-26] 

Detailed  processes  used  by  labs  to  perform  evaluations  (beyond  what  is  in  the  Common 
Evaluation  Method)  are  proprietary  and  may  be  hidden  from  the  public  and  other  labs. 

4.2.2.6  Assumptions  About  Policies  Concerning  Evaluations 

This  group  of  assumptions  addresses  national  and  agency  policies  relating  to  evaluations. 

Only  IA-Enabled  Products  [AN-27] 

National  policies  identify  only  IA  and  IA-enabled  products  as  subjects  for  evaluation. 
This  assumes  many  products  not  normally  considered  to  be  IA  or  IA-enabled,  such  as 
web  browsers,  image  viewers,  and  word  processors,  which  are  directly  connected  to  net¬ 
works,  have  no  vulnerabilities  or  security  impacts.  Many  of  these  products  have  pro¬ 
vided  unintended  system  access  to  attackers. 

4 . 3  The  NIAP  Responsibilities  Beyond  CCEVS 

NIAP’s  charter  includes  several  tasks  beyond  setting  up  and  operating  the  evaluation  and 
validation  scheme.  There  are  two  broad  areas  of  additional  responsibilities:  research  and 
development,  and  education  and  training.  The  first  is  an  explicit  requirement  of  the  NIAP 
letter  of  partnership.  The  second  is  a  derived  requirement  as  part  of  an  overall  program 
of  security  product  evaluations. 

4.3.1  Research  and  Development 

Three  areas  of  research  and  development  were  identified  as  goals  of  the  original  NIAP 
charter:  techniques  for  security  requirements  definition;  testing  methods,  tools,  and  tech¬ 
niques;  and  assurance  metrics. 

4.3. 1.1  Security  Requirements  Definition 

Current  methods  for  specifying  information  security  requirements,  in  terms  of  the  sensi¬ 
tivity  of  information,  confidentiality,  access  controls  and  other  protection  mechanisms, 
availability,  integrity,  and  fallback  and  recovery  mechanisms  are  incomplete  and,  because 
no  standards  exist,  are  often  inconsistent.  While  several  attempts  have  been  made  to  de¬ 
velop  taxonomies  of  security  requirements  (outside  the  NIAP),  this  remains  an  open  re¬ 
search  problem.  Fostering  and  promoting  consortium  development  of  PPs  is  needed. 

4.3. 1.2  Testing  Methods,  Tools,  and  Techniques 

The  NIAP  has  sponsored  projects  such  as  the  Common  Criteria  Toolbox.  This  tool  is  for 
evaluation  development,  however,  not  for  improving  IA  code  development  or  testing. 
Budget  priorities  have  excluded  development  of  tools  that  would  contribute  to  production 
of  more  secure  products. 

4.3. 1.3  Metrics 

Little  research  has  been  funded  into  metrics  for  information  assurance,  the  efficacy  of  IA 
methods  and  techniques,  or  the  return  on  investments  made  to  improve  IT  system  secu¬ 
rity.  For  example,  a  primary  metric  for  all  evaluations  would  be  the  value  of  the  informa- 
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tion  protected  or  the  losses  that  may  occur  with  security  breaches.  Neither  of  these  is 
generated  and  tracked.  Also,  the  cost  of  product  evaluation  is  treated  as  laboratory  pro¬ 
prietary  and  the  data  is  not  collected.  Some  data  is  being  collected  on  products  improve¬ 
ment  due  to  evaluation  but  is  insufficient  at  this  point. 

4.3.2  Education  and  Training 

Education  and  training  is  a  derived  requirement  as  part  of  an  overall  program  of  security 
product  evaluations.  The  NIAP  has  developed  training  materials  for  evaluators  and  stan¬ 
dards  for  product  documentation.  A  major  area  of  information  assurance  education  and 
training  that  has  not  been  adequately  addressed  is  the  education  and  training  of  IT  system 
purchasers,  operators,  and  users.  The  assumption  of  knowledgeable  consumers  described 
earlier  (Sec.  4. 2. 1.2)  is  not  supported  in  practice. 

4 . 4  Growth  of  Evaluation  Business 

Figure  6  shows  the  growth  in  the  number  of  evaluations  that  have  been  completed  and 
that  were  in  process  under  NIAP’s  supervision  over  the  past  four  years.  The  certificates 
issued  data  are  cumulative,  to  date.  The  in-evaluation  data  are  a  snapshot  for  the  year  and 
may  represent  some  products  that  are  carry-over  from  year-to-year,  and  some  products 
that  have  not  and  may  never  receive  a  certificate  of  completion.  These  statistics  while 
less  than  ideal  do  serve  to  show  a  growth  rate  in  the  obligation  of  the  “oversight”  provi¬ 
sions  that  are  required  by  the  Common  Criteria.  The  growth  rate  is  close  to  100  percent  - 
doubling  every  year  -  and  shows  no  signs  of  leveling  off. 
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Figure  6.  Growth  in  the  number  of  evaluations  conducted  under  the  NIAP 

The  original  evaluation  labs  have  expanded  their  business  to  match  this  growth.  There 
are  now  eight  certified  labs  performing  evaluations.  The  difficulties  of  staffing  this  rate 
of  business  growth  with  qualified  personnel  have  been  raised  as  an  issue  .  An  even  big¬ 
ger  problem  has  been  the  government’s  difficulty  in  staffing  validators  to  monitor  the 
progress  of  each  evaluation.  Originally,  going  as  far  back  as  the  TPEP  program  in  the 
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cf.  Sec.  5.3.4,  Evaluation  Personnel  and  Lab  Expectations  and  Observations. 
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1 980’s,  validators  were  drawn  from  government  staff  and  from  independent  Federally 
Funded  Research  and  Development  Centers  (FFRDC’s).  These  sources  have  not  been 
able  to  keep  up  with  the  increased  demand.  Several  structural  changes  have  kept  the 
backlog  to  a  minimum  and  the  NIAP  is  working  diligently  to  keep  up  with  the  growth. 


The  rapid  growth  in  evaluations  has  also  drawn  attention  and  a  resource  away  from  the 
NIAP’s  other  responsibilities  for  research  and  development,  protection  profiles,  and  edu¬ 
cation  and  training.  The  NIAP’s  funding  has  not  increased  to  cover  the  additional  obliga¬ 
tions  imposed  by  evaluation  growth.  Research  into  information  assurance  metrics,  for 
example,  has  not  provided  a  metric  and  tracking  system.  NIST  started  work  on  testing 
and  analysis  tools,  but  has  discontinued  it.  NSA  has  developed  PP's  for  military  needs, 
but  nobody  has  stepped  forward  to  produce  PP's  for  broader  government  and  commercial 
environments.  NSA  developed  training  materials  on  the  Common  Criteria.  NIST  devel¬ 
oped  training  materials  for  evaluators  and  validators,  but  infonnation  assurance  education 
and  training  for  the  broader  community  of  IT  system  owners  and  users  has  not  been  ad¬ 
dressed.  This  was  not  an  original  requirement  of  the  partnership,  but  is  needed  for  the 
consumer  side  of  product  evaluation. 

4 . 5  Findings  and  Conclusions 

Finding  [FN-1] 

NSA  and  NIST,  without  separate  funding  earmarked  for  the  NIAP,  have  produced  a  flexi¬ 
ble,  capably  staffed,  although  under-funded  product  evaluation  system. 

Finding  [FN-2] 

Budgeting  restrictions  have  prevented  the  NIAP  from  developing  education  and  training 
resources  for  IT  system  consumers,  tools  to  support  secure  product  development,  and 
protection  profiles  for  non-military  applications. 

Finding  [FN-3] 

Oversight  of  evaluations  has  been  limited  due  to  stretched  NIAP  budgets  and  a  shortage 
of  qualified  validators.  The  oversight  mechanism  meant  to  ensure  that  evaluations  con¬ 
form  uniformly  to  all  CC  requirements  is  under-funded.  The  NIAP  has  made  adjustments 
in  both  organization  and  resources.  The  current  number  of  evaluations  stresses  the  limits 
of  the  validation  resources  and  the  number  of  evaluations  continues  to  grow. 

Recommendation  [RN-1] 

Addressing  the  NIAP's  shortcomings  requires  giving  the  NIAP  a  more  substantial  footing 
organizationally  and  sufficient,  stable  funding  to  achieve  its  tasks.  While  there  is  some 
need  for  evaluation  scheme  improvements  (see  Chapter  5  on  stakeholder  expectations), 
there  is  no  need  to  start  the  process  over  from  scratch.  Revamping  the  administrative 
structure  processes,  and  expertise  needed  to  create  an  alternate  product  evaluation  scheme 
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would  be  costly  and  time  consuming,  and  there  is  little  evidence  that  a  more  successful 
result  would  be  achieved. 

Finding  [FN-4] 

Evaluations  take  longer  than  anticipated.  Evaluation  schedules  are  often  extended  be¬ 
yond  their  original  plan. 

Finding  [FN-5] 

Evaluations  frequently  result  in  modified  products  or  claims.  Most  evaluations  take 
longer  than  anticipated  either  because  the  product  does  not  satisfy  the  initial  claims,  or 
because  the  documentation  is  not  adequate.  The  result  is  that  either  the  product  or  the 
claims  must  be  modified.  This  is  actually  a  good  thing  if  one  ascribes  to  the  “truth  in  ad¬ 
vertising”  approach,  and  reduction  of  claims  in  marketing  materials  would  follow.  How¬ 
ever,  sufficient  data  gathering  has  not  been  done  to  adequately  quantify  this  effect.  It 
appears  (from  experienced  evaluators  and  validators  that  the  evaluation  documents  are 
the  primary  place  that  claims  are  reduced.  Ideally,  the  product  would  be  modified  to 
meet  the  claims,  but  it  is  usually  easier  to  obtain  a  certificate  by  reducing  claims.  The 
NIAP  is  working  to  have  claim  adjustments  documented,  but  this  only  helps  sophisticated 
customers  who  read  it.  When  conformance  to  a  PP  is  cited  or  required,  the  claims  in  the 
ST  cannot  be  reduced  below  those  of  the  PP.  DoD  requirements  for  PP  conformance 
should  be  continued. 

Finding  [FN-6] 

Developers  produce  large  amounts  of  data  relevant  to  evaluations  during  their  develop¬ 
ment  and  testing.  Only  a  small  portion  of  this  data  is  provided  to  evaluators  in  the  form 
of  evidence.  Consumers  typically  see  only  the  product’s  evaluation  certificate,  which 
contains  no  vulnerability  information,  and  the  other  infonnation  available  to  them  is  writ¬ 
ten  in  precise  evaluation  language,  and  typically  has  little  infonnation  about  residual  vul¬ 
nerabilities.  The  Education,  Training  and  Awareness  (ETA)  have  lagged  so  that  consum¬ 
ers  are  generally  not  well  educated  in  reading  the  evaluation  reports. 

Recommendation  [RN-2] 

Vulnerability  analysis  and  testing  results  from  product  evaluations  should  be  made  avail¬ 
able  for  system-level  C&A.  This  availability  can  be  through  the  laboratories  or  the  de¬ 
velopment  of  sanitized  reporting  methods. 
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5 .  Perceptions  of  Issues,  Problems,  and  Expectations 


This  is  the  third  in  a  set  of  three  independent  analyses  of  the  Cybersecurity  landscape  and 
NIAP  as  it  fits  into  that  landscape.  This  part  of  the  study  solicited  perceptions  about 
NIAP  issues,  problems,  and  expectations  from  cybersecurity  stakeholders.  It  is  important 
to  understand  what  stakeholders  believed  the  NIAP  was  doing  and  how  well,  and  com¬ 
pare  those  beliefs  to  The  NIAP's  objectives  and  observable  perfonnance.  Perception,  by 
definition  is  tricky,  because  it  means  that  no  two  people  see  a  problem  in  exactly  the 
same  way.  To  balance  individual  perceptions  and  shed  light  on  problems  perceived  by 
different  parts  of  the  information  technology  community,  stakeholders  were  grouped  into 
classes.  The  categories  set  up  were  intended  to  cover  the  entire  community.  The  sample 
is  not  statistically  significant  (not  a  distribution  based  sampling),  but  is  representative  of 
the  community  as  a  whole,  and  was  used  to  raise  issues. 

The  principal  data  collection  tool  was  the  personal  interview,  although  additional  data 
were  gleaned  from  written  material,  personal  communications,  and  team  member  experi¬ 
ences.  On  October  22,  2004,  IDA  hosted  a  Forum  to  solicit  broader  input.  At  this  Forum, 
individuals  representing  all  stakeholder  classes  were  invited  to  participate  in  a  review  of 
the  data  gathered  to  date  and  provide  input  as  to  additional  issues,  problems,  and  expecta¬ 
tions.  Finally,  a  notice  soliciting  additional  input  was  placed  in  the  Federal  Register. 
This  chapter  synthesizes  the  inputs  from  all  of  these  sources. 

The  first  section  of  this  chapter  describes  our  data  collection  processes.  The  second  sec¬ 
tion  describes  the  analysis  process  used  to  synthesize  the  results.  This  led  to  the  identifi¬ 
cation  of  16  different  topic  areas  that  were  of  concern  to  stakeholders.  The  third  section 
presents  the  expectations,  observations,  and  findings  derived  for  each  of  these  topic  areas. 

5 . 1  Data  Collection  Processes 

This  section  describes  our  interview  process,  as  well  as  the  other  mechanisms  used  for 
data  collection. 

5.1.1  Stakeholder  Classes 

To  ensure  thorough  data  collection  coverage  of  an  expected  wide  range  of  perceptions 
about  the  NIAP,  the  analysis  team  divided  the  population  of  IA  stakeholders  into  the  fol¬ 
lowing  categories: 

1 .  Department  of  Defense  (DoD)  -  individuals  in  the  DoD  who  represent  the  as¬ 
sured  information  system  customer  base. 

2.  Federal  Government  (FEDNonDoD)  -  individuals  outside  of  the  DoD  but  in  the 
Federal  government  who  represent  the  customer  base  (such  as  NASA  or  the 
FA  A). 
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3.  Process  -  individuals  who  are  or  have  been  involved  in  executing  the  current 
NIAP  process,  including  validators  and  lab  personnel,  as  well  as  NIST  and  NSA 
personnel. 

4.  Producers  (Large  and  Small)  -  developers  of  I A  or  IA-enabled  software  that 
may  be  subjected  to  evaluation  requirements,  including  large-scale  producers  such 
as  Microsoft,  IBM,  and  Oracle,  as  well  as  small  business  concerns. 

5.  Governance  -  individuals  who  are  instrumental  in  making  policy  and  mandating 
requirements  for  their  agencies,  such  as  heads  of  NSA  and  the  NIAP  or  Federal 
Agencies 

6.  Defense  Critical  -  individuals  who  are  involved  with  the  operational  capabilities 
of  the  commands  of  the  anned  services.  As  separate  from  branches  of  govern¬ 
ment  such  as  NASA,  FA  A,  etc. 

7.  Intelligence  -  individuals  who  are  involved  in  intelligence  gathering  activities. 

Several  of  these  categories  are  specialized  and  crosscut  the  larger  categories  along  differ¬ 
ent  dimensions.  The  individuals  were  assigned  to  the  category  that  most  closely  fit  their 
responsibilities.  (Several  people  represented  multiple  categories,  but  their  input  was 
counted  only  once.)  The  purpose  of  setting  up  these  categories  was  to  ensure  that  some 
coverage  of  input  from  these  perspectives  within  the  community  was  solicited.  They  did 
not  restrict  in  any  way  the  topics  discussed  or  the  issues  raised  either  in  the  interviews,  at 
the  Forum,  or  in  response  to  the  Federal  Register  announcement. 

As  discussed  in  Chapter  2,  the  Critical  Infrastructure  (Cl)  community  was  not  considered 
a  separate  stakeholder  class.  The  need  for  Cl  participation  in  considering  the  NIAP 
changes,  however,  was  raised  in  interviews  and  at  the  Forum.  There  are  two  issues  in  Cl, 
one  is  the  need  for  federal  mandates  or  guidance  and  the  right  of  commercial  entities  to 
determine  their  own  course  of  action.  The  Non-DoD  Federal  Government  category  cov¬ 
ers  the  Cl  community  for  the  purpose  of  collecting  input  for  this  study. 

The  data  collected  represents  a  sample  of  members  from  each  of  the  stakeholder  classes. 
The  intent  was  to  be  thorough  and  ensure  representative  sampling.  No  attempt  was  made 
to  justify  any  of  the  findings  by  statistical  analyses. 

5.1.2  Interview  process 

A  total  of  45  interviews  were  conducted.  There  was  no  attempt  to  get  a  statistically  sig¬ 
nificant  sample,  and  interviews  were  used  in  conjunction  with  other  source  data  in  a  “dis¬ 
covery”  process  for  developing  issues.  A  list  of  over  300  potential  interview  candidates 
across  all  stakeholder  classes  was  compiled.  In  March  2004,  100  selected  candidates 
were  mailed  letters  describing  the  NIAP  study  and  asking  for  participation  in  interviews. 
Selection  was  based  upon  their  specific  interest  in  product  evaluation  as  perceived  by  the 
analysis  team.  Based  on  responses  received,  interviews  were  scheduled  and  conducted. 
Additional  interviews  were  arranged  to  ensure  that  at  least  3  representatives  from  each 
stakeholder  class  were  interviewed. 
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Interviews  were  conducted  either  in  person  or  over  a  speakerphone.  Each  interview 
lasted  approximately  one  hour.  At  the  beginning  of  each  interview  it  was  explained  that, 
although  notes  were  being  taken  during  the  interview,  the  interview  itself  was  not  for  at¬ 
tribution.  The  interview  was  not  recorded.  The  interviewee  was  presented  a  brief  over¬ 
view  of  the  study’s  objectives  and  methodology.  The  initial  questions  covered  the  inter¬ 
viewee’s  current  responsibilities  to  determine  which  stakeholder  class  they  represented. 
Where  it  was  appropriate,  some  interviewees  were  assigned  more  than  one  stakeholder 
class.  Interviewees  were  asked  for  their  personal  opinions  and  perceptions  rather  than 
“official”  or  “corporate”  positions. 

Interviews  were  free  ranging  and  concentrated  on  areas  where  the  interviewee  exhibited 
some  expertise  or  expressed  an  opinion.  Each  interview  started  with  questions  on  general 
topics  in  order  to  draw  from  the  interviewee  issue  areas  of  interest  to  them.  Follow-up 
questions  addressed  specific  topics  and  issues  raised  by  the  interviewee.  Each  interview 
did  not  contain  the  same  questions,  only  topics,  and  the  purpose  was  to  raise  issue  areas 
not  conduct  an  opinion  poll.  Summary  sheets  were  produced  for  each  interview.  The  fol¬ 
lowing  is  a  sample  of  the  general  questions  used  in  the  NIAP  stakeholder  interviews: 

1 .  Are  you  familiar  with  the  NIAP  program/process? 

2.  Do  you  use  evaluated  products? 

3.  Are  you  aware  of  the  mutual  recognition  agreements  the  U.S.  has  with  evaluations 
performed  in  other  countries? 

4.  Are  you  satisfied  with  the  current  evaluation  process? 

5.  What  does  Assurance  mean  to  you? 

6.  What  does  a  certificate  mean?  What  should  a  certificate  mean? 

7.  How  do  C&A  and  product  evaluation  work  together  (or  not)? 

8.  How  much  should  the  process  of  evaluation/certification  cost?  How  should  the 
costs  be  paid?  Who  should  bear  the  costs? 

A  total  of  five  IDA  personnel  participated  in  conducting  the  interviews  with  at  least  two 
participating  in  each  interview.  This  served  to  ensure  that  all  essential  questions  were 
covered  and  that  notes  from  each  interview  were  complete.  The  IDA  personnel  were 
chosen  based  upon  their  knowledge,  background,  and  availability.  The  interviews  were 
then  analyzed  for  inclusion  in  the  study. 

5.1.3  Forum  Data  Collection 

On  October  22,  2004,  the  Institute  for  Defense  Analyses  conducted  a  NIAP  Forum  at  its 
Alexandria,  VA  facilities.  Over  150  people,  from  government  and  industry,  were  invited 
to  attend.  The  purpose  of  this  forum  was  to  provide  the  greater  NIAP  community  with  the 
opportunity  to  voice  their  issues,  concerns,  ideas,  and  complaints  regarding  the  NIAP 
process  as  it  pertained  to  them. 
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Approximately  45  people  representing  government,  industry,  the  NIAP  evaluation  labs, 
and  the  NIAP  validators  attended  this  forum.  The  initial  presentations  from  the  Institute 
for  Defense  Analyses  personnel  depicted  the  methodology  utilized  in  the  conduct  of  the 
NIAP  review.  After  this  brief  introduction,  the  floor  was  opened  to  audience  participation 
and  group  discussions  were  held  for  the  remainder  of  the  day. 

Areas  discussed  were: 

1)  The  Institute  for  Defense  Analyses  NIAP  Review  Overview. 

2)  The  NIAP  Policy  Overview  -  Statutes,  policies,  and  procedures  affecting  the 
NIAP. 

3)  Existing  NIAP  Practices  -  The  current  state  of  the  NIAP  and  how  it  has  evolved. 

4)  Stakeholder  Expectations/Data  Gathering  -  The  interview  process. 

5)  Other  topics  (no  topic  was  out  of  bounds) 

Considerable  feedback  was  received  from  the  attending  group  and  many  of  the  items  dis¬ 
cussed  were  captured  into  the  tables  of  expectations  in  section  5.3. 

5.1.4  Federal  Register  Announcement 

In  early  November  2004,  the  Institute  for  Defense  Analyses  (IDA),  in  consultation  with 
DHS  and  DoD  filed  a  Notice  in  the  Federal  Register  on  behalf  of  the  Department  of  De¬ 
fense  and  the  Department  of  Homeland  Security  to  provide  all  interested  parties  the  op¬ 
portunity  to  voice  their  concerns,  issues  and  ideas  on  the  NIAP  process.  This  was  a  last 
attempt  to  solicit  input  from  interested  parties  that  may  not  have  been  reached  by  the 
other  means  discussed  in  this  chapter. 

After  a  review  of  written  material  and  various  statutes  on  the  subject,  IDA  staff  attended 
a  briefing  on  the  issue  at  the  offices  of  DoD  Washington  Headquarters  Services  (WHS) 
on  December  8,  2004.  The  initial  advice  of  the  WHS  staff  was  to  approach  the  task  as  an 
information  collection  pursuant  to  the  requirements  found  in  the  Paperwork  Reduction 
Act  (PRA).  After  further  research  and  consideration,  it  was  proposed  to  WHS  that  the 
Notice  be  considered  a  Notice  for  General  Solicitation  of  Comments  and  not  a  collection 
under  PRA.  This  approach  was  approved. 

“The  General  Solicitation  of  Comments  from  the  General  Public  on  Review  of  the  Na¬ 
tional  Information  Assurance  Partnership  (NIAP)”  was  published  on  Wednesday,  Febru¬ 
ary  2,  2005  in  Federal  Register  Vol.  70,  No.  21,  page  5420.  The  notice  required  that 
comments  be  submitted  in  electronic  form  to  DoD/DHS  at  NIAPReview@ida.org  or  in 
written  form  to  the  Institute  for  Defense  Analyses  on  or  before  March  4,  2005. 

A  total  of  76  comments  were  received  and  processed  as  described  above. 

5.1.5  Literature  Search 

The  final  source  of  data  on  expectations  was  undertaken  as  a  literature  search  from  sig¬ 
nificant  parties  such  as  GAO,  The  National  Cyber  Security  Partnership  and  others  as  out¬ 
lined  in  Annex  A.  The  text  was  not  parsed,  but  conclusions  and  recommendations  in 
these  reports  were  gleaned  for  the  analysis.  A  total  of  18  additional  expectations  were 
added  to  the  analysis  list. 
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5 . 2  Analysis  Process 

The  notes  from  each  interview,  remark,  and  publication  were  translated  into  statements  of 
expectations  and  observations.  These  statements  were  collected  into  an  expectations  ma¬ 
trix,  which  was  used  to  synthesize  issues  across  all  stakeholder  classes.  The  rules  for 
synthesis  of  interviews  were  as  follows: 

•  Each  of  the  interviewers  agreed  that  the  final  set  of  expectations  and  observa¬ 
tions  recorded  from  each  interview  accurately  reflected  what  they  heard.  In 
many  cases,  the  statements  are  not  the  exact  words  expressed  by  interviewees 
or  in  written  material;  the  issues  were  paraphrased  for  accumulation  purposes. 

•  Statements  were  reviewed  for  appropriate  labeling,  consistent  terminology, 
and  characterization  as  an  expectation  or  an  observation. 

•  Similar  expectations  were  melded  within  the  interview  set  and  across  other 
forms  of  input. 

•  Unique  instances  of  expectations  not  corroborated  by  at  least  one  other  source 
were  considered  outliers  and  dropped.  (At  least  two  sources  were  needed  to 
include  the  result). 

•  Expectations  are  often  conflicting  and  reveal  a  basic  misunderstanding.  No 
harmonization  or  resolution  of  conflicting  expectation  was  undertaken  in  this 
chapter. 

•  Each  expectation  could  be  considered  a  finding,  however,  findings  were  re¬ 
served  for  synthesized  results  of  combinations  of  expectations  and  observa¬ 
tions.  As  a  result,  not  all  topic  areas  included  findings. 

This  process  resulted  in  a  total  collection  of  1093  statements  from  all  sources. 

5.2.1  Topic  Areas 

The  collection  of  statements  from  all  sources  were  then  categorized  and  assigned  the  fol¬ 
lowing  topic  areas  (topics  are  not  the  same  as  questions  asked  or  areas  covered  but  were 
derived  from  the  responses): 

1 .  Consumer  knowledge  and  understanding  of  evaluations 

2.  The  meaning  of  a  product  evaluation  certificate 

3.  Protection  profiles 

4.  Evaluation  personnel  and  evaluation  laboratory  issues 

5.  Testing  of  products  in  evaluations 

6.  Alternate  forms  of  assurance 
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7.  Relationship  between  C&A  and  product  evaluation 

8.  Mutual  recognition,  commercial  viability  and  related  issues 

9.  Research  areas 

10.  Target  of  evaluation  (TOE)  versus  product  evaluation 

1 1 .  Assurance  maintenance 

12.  Cost  and  time  issues 

13.  NSTISSP-1 1 

14.  Critical  infrastructure 

15.  Nefarious  and  malicious  behavior  in  code 

16.  Comments  concerning  NIST 

The  list  of  topics  areas  was  refined  iteratively  with  the  collected  expectations  and  obser¬ 
vations.  The  results  collected  for  each  of  these  areas  is  presented  in  the  sections  below. 

5 . 3  Expectations,  Observations,  and  Findings 

This  section  summarizes  the  expectations  and  observations  identified  in  stakeholder  in¬ 
terviews,  derived  from  written  material  provided  by  interviewees,  notes  taken  and  written 
inputs  received  at  the  NIAP  forum,  notes  taken  during  the  literature  review  and  inputs 
received  in  response  to  the  announcement  in  the  Federal  Register. 

Table  2  through  Table  17  documents  the  expectations  and  observations.  An  expectation 
was  associated  with  an  interviewee’s  delineation  of  how  the  system  should  work.  All 
others  were  tagged  as  observations.  Its  topic  area  and  a  sequence  number  identify  each 
expectation  or  observation  in  these  tables.  The  second  column  (description)  contains  the 
statement  of  expectation  or  observation.  The  third  column  shows  how  many  times  the 
expectation  or  observation  was  reported  during  interviews.  Since  interviewees  were  not 
proficient  in  all  areas,  and  follow-up  questions  were  restricted  to  expertise  or  opinion  ar¬ 
eas,  the  number  cannot  be  normalized,  but  does  provide  a  relative  strength  measure. 
Confirmation  of  these  issues  by  use  of  written  input  to  one  of  the  calls  for  data,  or  in  lit¬ 
erature  search  is  indicated  by  a  ♦.  In  the  case  of  no  numeric  entry,  the  issue  was  not 
raised  during  interviews  but  came  from  one  of  the  alternative  sources.  This  approach  was 
taken  because  interviews  allowed  for  follow-up  questioning  and  may  be  related  to  some 
of  the  other  sources.  The  additional  data  sources  were  not  classified  as  to  stakeholder 
classes  since  in  many  cases  the  stakeholder  classes  were  unknown.  The  remaining  col¬ 
umns  are  shaded  to  identify  the  stakeholder  class  or  classes  making  that  statement.  In 
some  cases,  the  number  of  stakeholder  classes  exceeds  the  number  of  interviewees  rais¬ 
ing  the  issue.  This  is  due  to  the  fact  that  some  interviewees  are  placed  in  more  than  one 
stakeholder  class,  indicated  by  an  asterisk  (*).  Findings  that  summarize  significant  ex- 
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pectations  are  presented  below  each  table  where  appropriate.  At  the  beginning  of  each 
section  we  summarize  the  general  observations  of  the  interviewers. 

5.3.1  Consumer  Knowledge  and  Understanding  of  Evaluations 

The  first  topic  area  is  the  consumer’s  knowledge  about  evaluation  processes  and  their  un¬ 
derstanding  of  evaluation  results.  Table  2  shows  a  broad  consensus  on  several  expecta¬ 
tions  and  observations.  The  first  entry  shows  the  strongest  agreement  (the  expectation 
expressed  by  the  largest  number  of  sources),  which  was  that  interpreting  current  evalua¬ 
tion  results  require  a  much  deeper  understanding  than  most  consumers  have.  At  the 
same  time,  a  large  number  of  sources  expressed  the  expectation  that  interpreting  evalua¬ 
tion  results  should  require  no  more  than  a  general  understanding  of  the  concepts.  Sources 
from  all  stakeholder  classes  observed  that  the  concept  of  assurance  as  embodied  in 
evaluations,  and  CC  terms  in  general  are  not  well  understood. 

Most  consumers  are  knowledgeable  about  their  systems  and  their  security  requirements 
(and  usually  have  deeper  expertise  available).  However,  current  evaluation  practices  re¬ 
quire  a  consumer  who  is  knowledgeable,  not  only  about  security,  but  also  in  a  number  of 
evaluation  nuances.  Not  many  consumers  have  the  necessary  expertise,  and  even  under 
ideal  circumstances  few  can  be  expected  to  gain  it.  Most  consumers  have  too  many  other 
complex  duties  to  become  experts  in  evaluation. 


Table  2.  Consumer  Knowledge  and  Understanding  Expectations  and  Observations 
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Expectations 

EKn-1 

Understanding  current  product  evaluation  results  requires  in-depth  knowledge 
of  evaluation  practices  and  the  arcane  terminology  of  the  CC  —  knowledge  that 
relatively  few  people  have. 

31  ♦ 

EKn-2 

People  with  only  a  general  understanding  of  evaluation  practices  and  minimal 
familiarity  with  CC  terminology  should  be  able  to  read  and  understand  evalua¬ 
tion  results. 

18 

EKn-3 

NIAP  currently  provides  training  for  evaluators.  NIAP  should  also  be  funded 
to  train  consumers  in  evaluation  processes  and  interpretation  of  results. 

15* 

EKn-4 

Education  is  a  responsibility  at  all  levels. 

2* 

Observations 

OKn-1 

The  concept  of  assurance  specifically  and  CC  tenns  in  general  are  not  well 
understood. 

15 
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OKn-2 

We  are  not  likely  to  get  many  consumers  who  fully  understand  evaluation  prac- 

7* 

tices  and  are  fluent  in  CC  terminology. 

OKn-3 

Not  every  consumer,  producer,  or  manager  has  to  be  an  evaluations  expert. 

One  knowledgeable  person  can  advise  a  group. 

OKn-4 

There  is  no  need  for  consumers  to  be  knowledgeable  about  evaluations. 

4 

Notes:  *  Indicates  the  number  of  stakeholder  classes  exceeds  the  number  of  interviews  raising  the  issue.  This  is  due  to  the  fact  that 
some  interviewees  are  placed  in  more  than  one  stakeholder  class. 

♦  Issue  identified  in  other  sources  (NIAP  Forum,  Federal  Registry  Announcement,  or  literature  search) 


Finding 

s  -  Consumer  Knowledge  and  Understanding 

FEKn-l 

Consumers  need  a  better  understanding  of  information  assurance  threats  and 
protection  methods,  and  a  basic  understanding  of  NIAP  evaluation  processes 
to  interpret  evaluation  results  and  make  informed  decisions  about  product 
suitability  for  their  needs. 

Supported  by  (table  above): 

Knowledge  and  Understanding  Expectation  1  [EKn-1] 

Knowledge  and  Understanding  Observation  1  [OKn-1] 

Knowledge  and  Understanding  Expectation  2  [EKn-2] 

FEKn-2 

Evaluations  are  often  reported  in  technical  CC  terms  and  do  not  state  in  plain 
language  what  information  assurance  protection  the  product  provides. 

Supported  by  (table  above): 

Knowledge  and  Understanding  Observation  1  [OKn-1] 

Knowledge  and  Understanding  Expectation  2  [Ekn-2] 

Knowledge  and  Understanding  Finding  1  [FEKn-l] 

5.3.2  Certificate  Meaning 

Given  that  consumers  are  not  evaluation  experts,  the  certificate  itself  needs  to 
make  the  information  it  conveys  more  easily  understood.  Table  3  shows  that  re¬ 
spondents  in  all  stakeholder  groups  thought  evaluation  certificates  should  accu¬ 
rately  characterize  the  types  of  protection  provided  by  the  product  and  how  it  was 
tested.  They  also  want  to  see  an  assessment  of  the  types  of  applications  for  which 
the  product  is  considered  suitable.  If  a  product  is  evaluated  as  a  firewall,  for  ex¬ 
ample,  then  the  evaluation  should  say  that  it  is  suitable  for  that  task.  In  most 
cases,  consumers  who  are  not  evaluation  experts  are  not  able  to  research  the  mate¬ 
rial  behind  current  evaluation  certificates  to  detennine  if  the  product  meets  their 
needs.  As  a  result,  many  equipment  buyers  in  the  DoD  consider  an  evaluation 
certificate  a  “check  box”  in  their  purchase  decisions,  required  independently  of 
the  protection  provided  by  the  product  or  its  suitability  for  the  intended  use. 
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The  meaning  of  a  certificate  would  also  be  enhanced  if  it  cited  the  product’s  con¬ 
formance  to  a  well-crafted  protection  profile  (PP).  (DoD  provides  for  this  by  re¬ 
quiring  the  ST  to  be  based  upon  an  approved  PP  where  one  exists.)  To  support 
this  objective,  each  such  PP  will  have  to  address  core  capabilities  of  a  relatively 
broad  class  of  products;  for  example,  firewalls.  A  further  improvement  in  certifi¬ 
cates  would  be  a  scheme  for  grading  the  security  provided  by  a  product  such  as 
home  use,  commercial  use,  or  national  security  use.  At  present,  certificate  de¬ 
scriptions  of  assurance  and  strength  of  function  are  meaningful  to  only  a  small 
number  of  evaluation  experts. 


Table  3.  Certificate  Meaning  Expectations  and  Observations 


Stakeholder  Catec 

i°ry 

Sequence  Num. 

Description 

Occurrences 

DoD 

Federal  Gov. 

Process 

Producer 

Governance 

Defense  Critical 

Intelligence 

Expectations 

ECm-1 

Evaluations  should  accurately  characterize  the  types  of 
protection  provided  by  the  product  and  how  it  was 
tested. 

23  ♦ 

ECm-2 

Evaluations  should  accurately  characterize  a  range  of 
suitable  uses  for  the  product. 

19 

ECm-3 

Certificates  should  identify  additional  documentation 
that  is  available  to  allow  knowledgeable  consumers  to 
assess  the  product’s  suitability  for  their  use. 

4 

1 

1 

Observations 


OCm-1 

Most  DoD  buyers  consider  a  certificate  the  essential 
factor,  independent  of  the  protection  provided  or  the 
product’s  suitability  for  use. 

7 

OCm-2 

Current  certificates  do  not  rate  a  product’s  claims  or 
suitability  for  use. 

24 

OCm-3 

if  we  had  protection  profiles  (PPs)  for  core  capabilities, 
certificates  of  compliance  could  provide  a  measure  of 
suitability  for  use. 

2*4 

OCm-4 

Certificates  should  not  attempt  to  rate  a  product’s  secu¬ 
rity  or  suitability  for  use. 

2* 

OCm-5 

The  consumer  wants  some  kind  of  independent  state¬ 
ment  that  states  the  goodness  of  the  product  (like  a  UL 
certificate). 

24 

OCm-6 

There  is  no  way  to  rescind  an  evaluation  certificate  for  a 
flawed  product 

♦ 

55 


Notes:  *  indicates  the  number  of  stakeholder  classes  exceeds  the  number  of  interviews  raising  the  issue.  This  is  due  to  the  fact  that  some 
interviewees  are  placed  in  more  than  one  stakeholder  class. 

♦  issue  identified  in  other  sources  (NIAP  Forum,  Federal  Registry  Announcement,  or  literature  search) 
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Findings  -  Evaluation  Certificates 

FECm-l 

Evaluation  certificates  in  general  do  not  identify  the  degree  of  security 
provided  by  the  product  or  provide  example  applications  for  which  the 
product  is  suitable. 

Supported  by  (table  above): 

Knowledge  and  Understanding  Finding  1  [FEKn-1] 

Certificates  Expectation  1  [ECm-1] 

Certificates  Expectation  2  [ECm-2] 

5.3.3  Protection  Profiles 

A  protection  profile  (PP)  is  a  set  of  specifications  (both  functional  and  assurance) 
against  which  a  product  may  be  evaluated.  Anyone  may  write  a  PP,  but  there  are 
no  requirements  for  evaluations  to  claim  conformance  to  any  PP  A  properly  vet¬ 
ted  PP  would  represent  the  requirements  of  a  class  of  stakeholders  for  a  product 
type  or  product  area.  NSA  developed  most  of  the  present  PPs  for  DoD  usage,  al¬ 
though  consortiums  have  also  developed  a  few  (such  as  SMARTCARD).  PP’s 
prevent  removing  claims  from  a  security  target  (ST)  and  ultimately  result  in  im¬ 
proved  security. 

As  Table  4  shows,  opinion  on  PPs  was  divided.  The  larger  group  believes  that 
conformance  to  PPs  should  be  required.  (DoD  provides  for  this  by  requiring  the 
ST  to  be  based  upon  an  approved  PP  where  one  exists.)  A  smaller  group,  how¬ 
ever,  did  not  want  to  require  confonnance  to  PPs.  Producers,  in  particular,  would 
prefer  not  to  be  held  to  standard,  vetted  protection  profiles.  Other  stakeholders 
also  expressed  the  opinion  that  verifying  TOE  claims  (the  current  alternative  to 
PPs)  is  adequate. 

Including  testing  requirements  in  PPs  was  surprisingly  popular,  except  among  the 
DoD  stakeholder  class.  Adding  test  requirements  to  a  PP  would  be  additional 
work  for  the  writers  of  PPs.  A  uniform  set  of  tests,  however,  would  make  testing 
much  more  consistent  across  evaluation  laboratories. 

Focusing  PPs  on  core  capabilities,  and  avoiding  non-essential  (proprietary)  fea¬ 
tures,  would  allow  a  wider  range  of  products  to  meet  the  conformance  require¬ 
ments. 
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Table  4.  Protection  Profiles  Expectations  and  Observations 


Stakeholder  Category 


equence  Num 

Description 

Occurrences 

DoD 

Federal  Gov. 

Process 

Producer 

Governance 

efense  Critica 

Intelligence 

w 

a 

Expectations 


EPP-1 

Conformance  to  one  or  more  trusted  PPs  should  be  required. 

204 

EPP-2 

Protection  profiles  should  include  a  definition  of  the  testing  re¬ 
quired  to  satisfy  conformance. 

20 

EPP-3 

Protection  profiles  for  core  capabilities  should  be  developed  to 
avoid  PP  proliferation. 

74 

EPP-4 

Demonstrating  product  conformance  to  one  or  more  protection 
profiles  should  not  be  mandated.  Verifying  TOE  claims  is  suffi¬ 
cient. 

5 

Observations 


OPP-1 

Protection  profiles  should  be  used  as  a  means  to  consolidate 
community  interests,  expertise,  and  support. 

204 

OPP-2 

Conformance  to  a  protection  profile  serves  as  a  measure  of  suit¬ 
ability  for  use.  PPs  can  also  aid  product  design. 

7 

OPP-3 

NIST  is  better  equipped  to  get  a  consensus  PP. 

64 

OPP-4 

Poorly  conceived  and  unstable  PPs  have  contributed  to  a  reluc¬ 
tance  to  develop  and  use  them. 

54 

OPP-5 

The  PP  needs  to  be  produced  by  the  government  and  widely  vet¬ 
ted. 

4 

OPP-6 

NSA  is  the  proper  entity  to  write  PPs  for  government  use. 

3 

OPP-7 

PP’s  contain  useful  information  to  state  requirements,  but  do  not 
work  as  evaluation  criteria. 

2, 

OPP-8 

PP  development  process  should  be  a  lot  faster. 

2 

OPP-9 

Producer  has  paid  lab  to  develop  a  core  capabilities  PP  for  their 
own  internal  use. 

2 

OPP-1 0 

PPs  should  not  be  wish-lists 

4 

Notes:  *  indicates  the  number  of  stakeholder  classes  exceeds  the  number  of  interviews  raising  the  issue.  This  is  due  to  the  fact  that  some  interviewees 
are  placed  in  more  than  one  stakeholder  class. 

♦  issue  identified  in  other  sources  (NIAP  Forum,  Federal  Registry  Announcement,  or  literature  search) 
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Findings  -  Protection  Profiles 

FEPP-l 

Protection  profiles  covering  core  infonnation  assurance  capabilities  for  gen¬ 
eral  use  have  not  been  developed.  A  number  of  protection  profiles  that  ad¬ 
dress  the  higher  levels  of  assurance  for  national  security  systems  have  been 
developed  by  NSA  for  use  by  that  community.  Protection  profiles  for  capa¬ 
bilities  that  satisfy  more  modest  assurance  requirements  have  not  been  de¬ 
veloped. 

Supported  by  (tables  above): 

Certificates  Finding  2  [EQn-2] 

Protection  Profiles  Expectation  4  [EPP-4] 

5.3.4  Evaluation  Personnel  and  Lab  Expectations  and  Observations 

The  principal  issues  regarding  evaluation  personnel  and  laboratories  were  the  ap¬ 
parent  conflict  of  interest  in  labs  preparing  evidence  and  conducting  the  evalua¬ 
tion  for  the  same  product,  and  certification  of  evaluators  (see  Table  5).  One  group 
thought  labs  should  be  restricted  from  preparing  evidence  for  the  evaluations  they 
conduct.  Another,  roughly  equal-sized  group  thought  that  labs  could  keep  these 
responsibilities  separate  and  ensure  there  was  no  conflict  of  interest. 

Certification  of  evaluation  personnel  was  recommended  as  a  means  to  gain  con¬ 
sistency  in  evaluations  across  laboratories.  A  certification  program  is  not  avail¬ 
able  and  needs  to  be  developed.  The  perceived  high  turnover  at  labs  was  cited  as 
contributing  to  inconsistency  .  Certification  of  validators  overseeing  evaluations 
was  also  recommended.  Calling  for  U.S.  citizenship  does  not  seem  reasonable  in 
light  of  Mutual  Recognition  Agreements  (MRAs). 

Process  stakeholders  thought  that  government  oversight  was  intrusive.  In  contrast 
to  other  accreditation  programs  that  NIST  and  NVLAP  run,  this  is  the  first  stan¬ 
dard  that  requires  such  high  degree  of  government  oversight  in  detail  for  each 
evaluation.  The  Cryptographic  Module  Validation  Program  (CMVP,  FIPS  140) 
provides  some  government  oversight,  but  not  to  the  extent  CCEVS  requires.  This 
level  of  oversight  should,  however,  contribute  to  evaluation  consistency. 


21  While  high  turnover  rate  was  discussed  among  several  interviewees,  we  do  not  have  data  to  support  the 
contention. 
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Table  5.  Evaluation  Personnel  and  Lab  Expectations  and  Observations 


S 

s 

Z 

QJ 

QJ 

fi 

Q-» 

= 

Oi 

C/3 


Description 


Stakeholder  Category 


Expectations 

EPe-1 

There  is  an  apparent  conflict  of  interest  between  preparing  evidence 
for  an  evaluation  and  conducting  an  impartial  evaluation.  Labs 
should  not  prepare  evidence  for  evaluations  they  conduct. 

11 

EPe-2 

Personnel  performing  evaluations  should  be  formally  certified  as 
NIAP  evaluators. 

12 

EPe-3 

Personnel  overseeing  evaluations  should  be  formally  certified  as 
NIAP  validators. 

11 

EPe-4 

A  single  lab  can  provide  both  evidence  and  the  evaluation,  but  they 
need  to  separate  responsibilities  so  a  conflict  of  interest  does  not 
exist.  (The  amounts  of  talent  and  market  impact  were  cited  as  is¬ 
sues). 

10 

EPe-5 

Evaluation  results  must  be  repeatable  across  labs  and  among  prod¬ 
ucts. 

6 

EPe-6 

There  should  be  no  appearance  of  conflict  of  interest  in  any  aspect  of 
evaluations. 

5 

EPe-7 

U.S.  labs  should  require  U.S.  citizenship  for  evaluation. 

4 

Observations 

OPe-1 

Oversight  is  intrusive,  delegate  responsibility. 

4 

OPe-2 

The  evaluation  process  is  not  consistent  and  repeatable  between  labs. 

34 

OPe-3 

High  personnel  turnover  rates  at  labs  degrade  evaluation  perform¬ 
ance. 

3 

C3 
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Notes:  *  indicates  the  number  of  stakeholder  classes  exceeds  the  number  of  interviews  raising  the  issue.  This  is  due  to  the  fact  that  some  inter¬ 
viewees  are  placed  in  more  than  one  stakeholder  class. 

♦  issue  identified  in  other  sources  (NIAP  Forum,  Federal  Registry  Announcement,  or  literature  search) 
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Findings  -  Evaluation  Personnel 

FEPe-l 

Product  evaluators  come  from  a  variety  of  disciplines  with  varying  levels  of 
expertise.  Although  NIAP  checks  that  evaluation  processes  are  followed 
correctly,  no  process  has  been  established  to  ensure  adequate  training  of 
evaluators  and  validators. 

Supported  by  (table  above): 

Evaluation  Personnel  Expectation  2  [EPe-2] 

Evaluation  Personnel  Expectation  3  [EPe-3] 

Evaluation  Personnel  Expectation  5  [EPe-5] 

Evaluation  Personnel  Observation  2  [OPe-2] 

Evaluation  Personnel  Observation  3  [OPe-3] 

FEPe-2 

Current  conflict  of  interest  rules,  particularly  those  that  allow  laboratories  to 
develop  evidence  and  conduct  evaluations  on  the  same  products,  are  open  to 
potential  abuse. 

Supported  by  (table  above): 

Evaluation  Personnel  Expectation  1  [EPe-1] 

Evaluation  Personnel  Expectation  6  [EPe-6] 

5.3.5  Testing  of  Products  in  Evaluation 

There  was  general  agreement  that  product  evaluations  need  to  include  more  test¬ 
ing  to  meet  the  security  assurances  sought.  DoD  Protection  Profiles  often  add  the 
AVA_VLA.3  assurance  requirement  (called  plus-up  by  several  interviewees)  to 
increase  the  amount  of  vulnerability  testing  being  perfonned.  Expectations  on 
providing  tools  and  requiring  source  code  review  came  from  all  stakeholder 
classes  (see  Table  6).  Automated  tools  find  many  common  mistakes  that  can  oc¬ 
cur  during  software  coding  (open  ports,  buffer  overflows,  obvious  Trojan  horses, 
and  back  door  entries).  In  order  for  tools  to  have  maximum  value,  they  need  to  be 
standardized,  portable,  and  freely  available.  Some  process  of  certification  by  the 
laboratories  needs  to  be  developed  to  ensure  that  the  current  and  certified  copies 
of  tools  are  used  in  evaluations.  However,  general  open  availability  would  en¬ 
courage  the  developers  to  use  the  tools  before  evaluation,  thus  shortening  evalua¬ 
tion  iterations  and  saving  time. 

Automated  source  code  review  is  more  reliable  than  testing  at  finding  buffer  over¬ 
flows  and  some  forms  of  malicious  code,  as  well  as  apparently  benign  back  doors 
left  by  programmers.  To  avoid  the  potential  exposure  of  proprietary  intellectual 
property  in  source  code,  it  was  suggested  that  producers  run  their  own  (auto¬ 
mated)  source  code  review  and  submit  the  results  to  evaluators. 
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A  small  minority  suggested  that  they  would  prefer  making  it  easier  to  conduct 
evaluations,  encouraging  evaluation  of  more  products,  rather  than  making  it 
harder  by  increasing  testing  requirements.  In  fact,  tool  usage  could  be  applied  to 
products  that  are  not  normally  thought  of  as  IA  or  IA-enabled  but  have  an  impact 
on  system  security. 


Table  6.  Testing  of  Products  in  Evaluation  Expectations  and  Observations 


Stakeholder  Category 
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ETe-1 

A  sanctioned  set  of  automated  analysis  tools  is  needed  to 
increase  trust  in  evaluation  results.  These  tools  should  be 
funded  by  the  government  and  made  available  to  all. 

234 

: 

□ 

ETe-2 

Product  evaluation  at  all  levels  must  include  source  code 
review. 

12 

ETe-3 

All  evaluations  should  include  vulnerability  testing. 

74 

_ 

_ 

ETe-4 

The  CEM  should  specify  testing  requirements  in  detail  and 
provide  example  test  cases  for  each  functional  requirement. 

6 

ETe-5 

Evaluations  below  EAL4  need  to  include  more  testing. 

84 

z 

ETe-6 

Producers  should  be  required  to  provide  automated  source 
code  review  as  evidence. 

3 

Observations 


OTe-1 

Adding  testing  requirements  to  evaluations  below  EAL4  is 
not  necessary. 

3 

OTe-2 

Evaluations  should  not  require  that  labs  be  given  access  to 
the  producer’s  source  code. 

3 

OTe-3 

Testing  in  evaluations  needs  to  be  automated. 

24 

OTe-4 

Moving  testing  requirements  down  to  lower  EAL  levels  is  not 
the  right  thing  to  do.  Evaluating  more  products  to  raise  the 
overall  level  of  assurance  is  more  appropriate. 

2* 

OTe-5 

Automated  testing  tools  would  raise  the  confidence  and  trust 
in  the  results  produced. 

2. 

OTe-6 

Evaluations  below  EAL5  are  a  waste  of  time. 

2 

Notes:  *  indicates  the  number  of  stakeholder  classes  exceeds  the  number  of  interviews  raising  the  issue.  This  is  due  to  the  fact  that  some  inter¬ 
viewees  are  placed  in  more  than  one  stakeholder  class. 

♦  issue  identified  in  other  sources  (NIAP  Forum,  Federal  Registry  Announcement,  or  literature  search) 
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Findings  -  Testing 

FETe-l 

Automated  tools  can  help  standardize  evaluation  processes,  perform  more 
thorough  product  analyses,  and  reduce  evaluation  costs.  No  standard  collec¬ 
tion  of  automated  security  analysis  tools  has  been  developed  or  assembled  to 
support  evaluations. 

Supported  by  (tables  above): 

Testing  Expectation  1  [ETe-1] 

Testing  Observation  3  [OTe-3] 

Testing  Observation  5  [OTe-5] 

Knowledge  and  Understanding  Finding  2  [FEKn-2] 

FETe-2 

Both  the  Common  Evaluation  Method  (CEM)  and  protection  profiles  often 
omit  detailed  testing  requirements. 

Supported  by  (tables  above): 

Testing  Expectation  3[ETe-3] 

Testing  Expectation  5  [ETe-5] 

Protection  Profile  Expectation  2  [EPP-2] 

FETe-3 

No  automated  review  of  source  code  is  required  for  evaluations  at  EAL-4 
and  below.  For  software  products,  the  code  represents  a  complete  technical 
specification  of  the  product's  functionality,  and  is  much  more  revealing  than 
the  other  design  and  implementation  documentation  that  is  considered  in 
evaluations.  Automated  source  code  review  could  screen  out  many  common 
security  flaws  that  currently  go  undetected. 

Supported  by  (table  above): 

Testing  Finding  1  [FETe-l] 

Testing  Expectation  2  [ETe-2] 

Testing  Expectation  6  [ETe-6] 

Testing  Observation  5  [OTe-5] 

5.3.6  Alternate  Forms  of  Assurance 

Table  7  shows  that  alternate  fonns  of  assurance  are  not  significant  concerns  for 
most  stakeholder  classes.  Alternate  forms  of  assurance,  however,  may  reduce 
costs  and  could  be  useful  at  lower  evaluation  assurance  levels.  Several  interview¬ 
ees  believed  that  alternative  assurance  methods  are  needed,  especially  to  reduce 
costs  (such  as  a  “CC  life”).  This  is  the  case  for  organizations  or  situations  that 
cannot  afford  to  pay  for  evaluations,  such  as  many  small  web  applications,  small 
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businesses,  and  open  source  software  (OSS)  projects.  Support  for  alternative  as¬ 
surance  levels  was  strongest  for  use  in  lower  assurance  evaluations.  Many  be¬ 
lieved  that  NIAP  evaluation  would  be  strengthened  if  the  alternative  assurance 
methods  were  used  to  supplement  the  NIAP  evaluation,  with  SSE,  CMM,  and 
CMMI  specifically  mentioned  as  examples  of  alternative  assurance  methods. 
These  assurance  methods  would  augment  (not  replace)  the  current  assurance 
methods.  These  alternatives  are  discussed  in  detail  in  Chapter  6. 


Table  7.  Alternate  Forms  of  Assurance  Expectations  and  Observations 


Stakeholder  Category 

Sequence  Num. 

Description 

Occurrences 

DoD 

Federal  Gov. 

Process 

Producer 

Governance 

Defense  Critical 

Intelligence 

Expectations 

EAa-1 

Alternate  forms  of  assurance  are  needed  (beside  current 
NIAP  evaluations)  to  reduce  costs  (for  example,  for  small 
businesses). 

7  ♦ 

1 

■ 

■ 

EAa-2 

Phase  in  CMM  and/or  CMMI  System  Security  Evaluations 
(SSE)  in  place  of  current  NIAP  processes. 

2 

EAa-3 

Need  a  Common  Criteria  Mite’  with  alternate  forms  of  assur¬ 
ance  and  less  cost. 

2  ♦ 

EAa-4 

Best  Practices  should  be  incorporated. 

♦ 

Observations 

OAa-1 

Supplementing  product  evaluation  with  alternate  forms  of 
assurance  (such  as  an  SSE  CMM/CMMI)  would  improve 
product  assurance. 

84 

OAa-2 

Alternate  forms  of  assurance  should  be  researched  and 
considered  in  evaluation.  Alternate  forms  of  assurance 
could  be  used  in  cases  where  evaluation  is  not  timely,  ap¬ 
propriate,  or  cost  effective. 

24 

1 

OAa-3 

Alternate  forms  of  assurance  are  not  needed  and  will  only 
cloud  the  issue. 

2 

Notes:  *  indicates  the  number  of  stakeholder  classes  exceeds  the  number  of  interviews  raising  the  issue.  This  is  due  to  the  fact  that  some  interview¬ 
ees  are  placed  in  more  than  one  stakeholder  class. 

♦  issue  identified  in  other  sources  (NIAP  Forum,  Federal  Registry  Announcement,  or  literature  search) 
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5.3.7  Relationship  between  Certification  and  Accreditation  (C&A)  and  Product 
Evaluation 

Certification  and  accreditation  of  systems  was  considered  essential  by  all  stakeholder 
classes,  whether  or  not  product  evaluations  are  involved.  Product  evaluations  can  help 
C&A  and  should  be  taken  into  account.  In  fact,  it  was  recommended  that  both  FISMA 
and  DITSCAP  C&A  guidance  be  revised  to  include  requirements  for  using  evaluated 
products  (see  Table  8). 

Table  8.  Relationship  between  Certification  and  Accreditation  (C&A)  and  Product  Evaluation 
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Expectations 


ECa-1 

C&A  for  security  systems  is  an  absolute  requirement. 

28 

ECa-2 

DITSCAP  and  FISMA  C&A  requirements  should  be 
modified  to  include  evaluated  products. 

14 

ECa-3 

DITSCAP  and  FISMA  should  impose  uniform  require¬ 
ments  for  the  use  of  evaluated  products. 

8 

ECa-4 

Product  evaluation  is  a  required  part  of  C&A. 

2 

ECa-5 

Product  evaluation  data  should  be  made  available  for 
C&A 

♦ 

Observations 

OCa-1 

Use  of  evaluated  products  should  add  value  to  C&A. 

10 

OCa-2 

A  good  solution  to  composability”  will  not  replace 
C&A  but  can  assist  it. 

5* 

OCa-3 

C&A  is  sufficient  by  itself.  Product  evaluations  add  no 
value. 

3 

OCa-4 

C&A  should  be  able  to  reuse  evidence  from  product 
evaluations,  allowing  C&A  to  concentrate  on  interfaces. 

2 

22  Composability  -  refers  to  the  problem  that  combining  products  with  well-established  security  properties 
can  produce  systems  with  significant  security  flaws.  How  to  detect  these  flaws  in  systems  built  from  se¬ 
cure  components  is  not  yet  solved. 
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Notes:  *  indicates  the  number  of  stakeholder  classes  exceeds  the  number  of  interviews  raising  the 
issue.  This  is  due  to  the  fact  that  some  interviewees  are  placed  in  more  than  one  stakeholder  class. 

♦  issue  identified  in  other  sources  (NIAP  Forum,  Federal  Registry  Announcement,  or  literature 
search) _ 


Findings  -  C&A 

FECa-l 

Certification  and  accreditation  of  systems  was  considered  essential  by  all 
stakeholder  classes,  and  product  evaluation  should  improve  C&A 

Supported  by  (table  above): 

C&A  Expectation  1  [ECa-1] 

C&A  Expectation  2  [ECa-2] 

C&A  Expectation  5  [ECa-5] 

C&A  Observation  1  [OCa-1] 

5.3.8  Mutual  Recognition,  Commercial  Viability  and  Related  Issues 

There  was  a  consensus  from  all  stakeholder  classes  on  the  need  for  mutual  recog¬ 
nition  agreements,  as  shown  in  Table  9.  Differences  in  evaluation  schemes  used 
in  different  countries,  which  lead  to  inconsistent  evaluations,  however,  were 
raised  as  an  issue.  Some  suggested  that  the  limit  on  current  agreements  (EAL4 
and  below)  could  be  raised  to  EAL  7. 

Opinions  on  the  importance  of  commercial  viability  of  evaluations  -  that  is,  suffi¬ 
cient  public  demand  for  evaluated  products  to  sustain  the  NIAP  evaluation  proc¬ 
ess  without  government  support  -  were  mixed. 

All  stakeholder  classes  thought  that  by  some  means  shifting  liability  for  cyber 
losses  away  from  the  consumer  would  help  the  commercial  viability  of  evalua¬ 
tions.  This  would  mean  that  producers,  evaluation  labs,  the  government,  or  some 
combination  would  have  to  stand  behind  evaluated  products  and  warrant  them 
against  certain  types  of  loss.  If  evaluations  do  not  provide  sufficient  added  assur¬ 
ance  to  reduce  consumers’  liabilities,  consumers  will  not  see  any  value  in  evalua¬ 
tion  processes  and  will  not  pay  any  extra  premium  for  evaluated  products. 
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Table  9.  Mutual  Recognition,  Commercial  Viability  and  Related  Topics  Expectations  and 

Observations 
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Expectations 


EMR-1 

Mutual  recognition  is  necessary. 

20  ♦ 

EMR-2 

Evaluations  must  be  consistent  (repeatable)  across 
labs,  including  labs  in  other  countries.  At  present,  dif¬ 
ferent  countries  have  adopted  different  evaluation 
schemes  and  these  are  not  seen  as  equivalent. 

94 

1 

EMR-3 

Commercial  viability  is  necessary. 

74 

EMR-4 

Commercial  viability  is  not  necessary. 

4 

Observations 

OMR-1 

Consumers  typically  have  to  assume  liability  for  any  in¬ 
formation  assurance  losses.  If  evaluated  products  came 
with  warranties  against  such  losses,  it  would  give  them 
a  commercial  advantage  over  unevaluated  products. 

9 

OMR-2 

Current  MRA’s  are  limited  to  EAL4  and  below.  This  limit 
should  be  raised  to  EAL  7. 

64 

OMR-3 

In  spite  of  MRA’s,  not  all  product  evaluations  at  EAL4 
and  below  are  accepted  country  to  country. 

24 

OMR-4 

There  is  no  need  to  change  EAL  limits  in  MRA’s  (up  or 
down).  The  limits  are  good  where  they  are. 

3 

OMR-5 

Under  the  current  paradigm  NIAP  evaluations  cannot  be 
made  commercially  viable. 

24 

Notes:  *  indicates  the  number  of  stakeholder  classes  exceeds  the  number  of  interviews  raising  the  issue.  This  is  due  to  the  fact  that  some  inter¬ 
viewees  are  placed  in  more  than  one  stakeholder  class. 

♦  issue  identified  in  other  sources  (NIAP  Forum,  Federal  Registry  Announcement,  or  literature  search) 
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Findings  -  Mutual  Recognition  Agreements 

FEMR-l 

NIAP  has  not  addressed  warranty  or  liability  issues  for  evaluated  products. 
No  legal  or  business-case  analyses  on  who  might  underwrite  warranties 
for  evaluated  products  was  found,  or  what  effect  warranties  might  have  in 
promoting  adoption  of  evaluated  products. 

Supported  by  (table  above): 

MRA  Observation  1  [OMR-1] 

FEMR-2 

Mutual  Recognition  is  necessary. 

Supported  by  (table  above): 

MRA  Expectation  1  [EMR-1] 

5.3.9  Research  Areas 

The  NIAP  as  originally  chartered  intended  to  support  tool  development,  research, 
and  evaluations  (see  Section  3.1).  The  burden  developing  evaluation  processes, 
setting  up  labs,  etc.,  has  become  so  great  that  few  resources  are  left  to  devote  to 
research  and  tools.  Sorely  lacking  are  metrics  that  both  quantify  security  aspects 
of  software  programs  and  the  effectiveness  of  evaluations.  Several  respondents 
suggested  that  the  NIAP’s  research  focus  should  be  restored  (see  Table  10). 

A  particularly  difficult  problem  is  how  to  combine  product  metrics  to  compute  a 
system’s  overall  security  posture.  This  problem  is  called  composability.  While 
composability  was  recognized  as  a  problem,  there  were  mixed  opinions  on 
whether  it  could  be  solved  or,  if  it  could  be  solved,  whether  the  solution  would 
have  sufficiently  wide  applicability  to  make  it  worth  the  effort. 


Table  10.  Research  Areas  Expectations  and  Observations 


Stakeholder  Category 

Sequence  Num. 

Description 

Occurrences 

DoD 

Federal  Gov. 

Process 

Producer 

Governance 

Defense  Critical 

Intelligence 

Expectations 

ERe-1 

NIAP  should  support  research  into  assurance  metrics,  com¬ 
posability,  and  return  on  investment. 

7. 

ERe-2 

Research  is  needed  to  develop  measures  of  product  assur¬ 
ance  and  evaluation  effectiveness  so  that  consumers  can 
compare  products  and  evaluation  results. 

4* 

Observations 
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ORe-1 

A  set  of  security  metrics  is  lacking. 

54 

: 

: 

: 

ORe-2 

Composability  is  solvable;  research  is  needed  to  produce  the 
solution. 

4 

ORe-3 

Composability  is  not  solvable  and  should  not  be  pursued. 

3 

Notes:  *  indicates  the  number  of  stakeholder  classes  exceeds  the  number  of  interviews  raising  the  issue.  This  is  due  to  the  fact  that  some  inter¬ 
viewees  are  placed  in  more  than  one  stakeholder  class. 


♦  issue  identified  in  other  sources  (NIAP  Forum,  Federal  Registry  Announcement,  or  literature  search) 


Findings  -  Research 

FERe-l 

A  number  of  open  research  problems  remain  unaddressed,  including  assur¬ 
ance  metrics  and  solutions  to  composability  among  other  security  problems. 

Supported  by  (table  above); 

Research  Expectation  1  [ERe-1] 

Research  Expectation  2  [ERe-2] 

Research  Observation  1  [ORe-1] 

5.3.10  Target  Of  Evaluation  (TOE)  Versus  Product  Evaluation 

An  expectation  expressed  by  all  stakeholder  classes  was  that  evaluations  should 
be  conducted  on  products  as  delivered  and  as  used  in  normal  environments,  not  on 
specially  configured  targets  of  evaluation  (TOEs).  The  products  are  what  con¬ 
sumers  buy  and  use,  not  the  TOEs.  A  smaller  group,  which  did  not  include  pro¬ 
ducers,  expressed  the  opinion  that  evaluating  TOEs  against  specific  claims  meets 
particular  needs  and  is  acceptable  if  you  understand  the  fine  print  in  the  certifi¬ 
cate. 
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Table  11.  TOE  Versus  Product  Evaluation  Expectations  and  Observations 


Stakeholder  Category 


Sequence  Num. 

Description 

Occurrences 

DoD 

Federal  Gov. 

Process 

Producer 

Governance 

Defense  Critical 

Intelligence 

Expectations 

ETOE-1 

Product  evaluation  must  be  for  the  product  as  delivered  (not 
TOE). 

12  ♦ 

Observations 

OTOE-1 

TOE  evaluations  are  acceptable  do  not  need  product 
evaluations. 

6 

Notes:  *  indicates  the  number  of  stakeholder  classes  exceeds  the  number  of  interviews  raising  the  issue.  This  is  due  to  the  fact  that  some  in¬ 
terviewees  are  placed  in  more  than  one  stakeholder  class. 


♦  issue  identified  in  other  sources  (NIAP  Forum,  Federal  Registry  Announcement,  or  literature  search) 


Findings  -  Targets  of  Evaluation 

FETOE-l 

A  number  of  products  have  been  evaluated  in  unusual  configurations 
and  environments  that  do  not  represent  consumers'  general  use.  These 
evaluations  do  not  provide  sufficient  information  to  determine  how  these 
products  will  perfonn  in  typical  system  configurations  and  normal  use. 

Supported  by  (tables  above): 

TOE  Expectation  1  [ETOE-1] 

Knowledge  and  Understanding  Expectation  1  [EKn-1] 

Knowledge  and  Understanding  Expectation  2  [EKn-2] 

Knowledge  and  Understanding  Observation  1  [OKn-1] 

Knowledge  and  Understanding  Finding  1  [FEKn-1] 

5.3. 1 1  Maintenance  Assurance 

Producers  often  create  new  releases  of  products  with  defect  corrections  and  minor 
functional  enhancements.  At  present,  the  standard  CC  assurance  packages  for 
EAL  1  through  7  have  no  provisions  for  accommodating  such  changes.  While 
maintenance  assurance  and  flaw  remediation  packages  are  part  of  the  standard, 
they  are  not  part  of  any  assurance  packages.  Deviations  from  the  standard  assur¬ 
ance  packages  are  seldom  made.  One  exception  is  the  case  where  PP  confor¬ 
mance  is  claimed  and  the  PP  requires  maintenance  assurance.  Each  new  release  is 
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therefore  treated  as  a  new  product,  requiring  full  evaluation.  A  number  of  stake¬ 
holders  felt  that  analysis  and  testing  of  minor  product  changes,  which  would  re¬ 
quire  considerably  less  effort  than  a  full  evaluation,  should  be  sufficient  to  extend 
the  original  certificate  to  cover  the  updated  product.  These  provisions  occur 
within  the  maintenance  assurance  packages.  Further,  it  is  expected  that  where  se¬ 
curity  issues  are  involved,  security  failures  and  the  patches  to  improve  the  product 
should  be  accompanied  by  notification  to  all  registered  users.  These  provisions 
occur  in  the  flaw  remediation  packages.  DoD  may  handle  this  in  individual  pro¬ 
curements  as  an  acquisition  issue  or  directly  in  the  PPs,  but  the  more  formal  main¬ 
tenance  assurance  and  flaw  remediation  would  provide  increased  uniformity  to 
DoD  and  a  measure  of  protection  to  all  stakeholders.  Both  the  CC  and  the  CMVP 
have  approaches  for  revalidation  that  address  the  type  of  change  to  a  product. 
They  simply  need  to  be  part  of  the  EAL  packages. 
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Table  12.  Assurance  Maintenance  Expectations  and  Requirements 


Stakeholder  Categor 

y 

Sequence  Num. 

Description 

Occurrences 

DoD 

Federal  Gov. 

Process 

Producer 

Governance 

Defense  Critical 

Intelligence 

Expectations 

EAM-1 

When  changes  are  made  to  a  product 
(either  for  feature  enhancement  or  to 
correct  defects),  a  process  is  needed  to 
validate  the  changes  and  extend  its  cer¬ 
tificate,  short  of  full  re-evaluation. 

104 

EAM-2 

Flaws  in  commercial  products  should  be 
looked  at  (flaw  remediation  should  be 
part  of  maintenance  assurance,  and 
both  should  be  required). 

♦  (strong 
input) 

Observations 

OAM-1 

Do  not  recommend  a  maintenance  as¬ 
surance  program. 

2 

Notes:  *  indicates  the  number  of  stakeholder  classes  exceeds  the  number  of  interviews  raising  the  issue.  This  is  due  to  the  fact  that  some  in¬ 
terviewees  are  placed  in  more  than  one  stakeholder  class. 

♦  issue  identified  in  other  sources  (NIAP  Forum,  Federal  Registry  Announcement,  or  literature  search) 

Findings  -Assurance  Maintenance 

FEAM- 

l 

Evaluations  should  include  both  maintenance  assurance  and  flaw  remediation 
work  packages. 

Supported  by  (table  above): 

Maintenance  and  Assurance  Expectation  1  [EAM-1] 

Maintenance  and  Assurance  Expectation  2  [EAM-2] 

5.3.12  Cost  and  Time  Issues 

Evaluation  costs  are  too  high  and  they  take  too  long.  These  are  common  com¬ 
plaints,  particularly  from  small  businesses.  The  documentation  generated  for 
evaluations  is  partly  responsible.  While  there  is  no  specific  documentation  listed 
in  the  CC,  information  presentation  and  content  result  in  the  development  of 
evaluation  specific  documentation.  Little  of  this  documentation  is  part  of  normal 
development  processes.  This  documentation  is  expensive  to  produce  and  often 
requires  revision  to  satisfy  evaluators.  While  this  documentation  may  be  required 
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at  higher  assurance  levels,  large  parts  of  the  information  required  are  present  in 
developer  documentation.  The  need  for  specific  documentation  is  not  present  in 
the  CC,  only  content.  Many  developers  are  either  convinced  to  provide  separate 
documentation,  or  prefer  to  do  so  for  proprietary  reasons.  The  CCEVS  should 
stress  that  content  is  the  important  part  of  the  documentation  process,  leaving  its 
interpretation  to  the  evaluation  laboratories.  The  labs  frequently  mandate  the  spe¬ 
cific  format/structure  of  the  documentation. 


Table  13.  Cost  and  Time  Issues  Expectations  and  Observations 
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Expectations 


ECT-1 

The  government  should  subsidize  evaluations  of 
products  that  are  used  in  highly  classified  applica¬ 
tions  and/or  have  limited  application. 

24 

ECT-2 

DoD  should  pay  for  the  product  evaluation  of  any 
product  it  is  considering  using  and  let  the  market 
benefit  from  that  expenditure. 

24 

ECT-3 

Evaluation  times  should  be  considerably  less  than 
product  release  cycles. 

4 

Observations 

OCT-1 

Evaluation  costs  are  too  high. 

144 

OCT-2 

Evaluations  take  too  long. 

94 

OCT-3 

Under  the  current  process,  small  companies  are  at  a 
real  cost  disadvantage  when  it  comes  to  product 
evaluations. 

74 

OCT-4 

Current  practices  produce  voluminous,  uninformative 
documentation  that  is  a  burden  to  all.  Documenta¬ 
tion  needs  to  be  written  in  plain  language  and 
streamlined. 

84 

OCT-5 

Evaluation  costs  are  acceptable. 

3 

OCT-6 

The  time  evaluations  take  is  acceptable. 

3 

OCT-7 

We  should  not  be  concerned  that  some  producers 
consider  evaluation  costs  a  barrier  to  entry.  They 
are  a  cost  of  doing  business. 

3 

: 

OCT-8 

The  government  should  help  small  businesses  with 
training  and  subsidies  for  evaluations. 

34 
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OCT-9 

Documentation  requirements  are  right  and  proper. 

2 

Notes:  *  indicates  the  number  of  stakeholder  classes  exceeds  the  number  of  interviews  raising  the  issue.  This  is  due  to  the  fact  that 
some  interviewees  are  placed  in  more  than  one  stakeholder  class. 


search) 


♦  issue  identified  in  other  sources  (NIAP  Forum,  Federal  Registry  Announcement,  or  literature 


5.3.13  NSTISSP-11 

Comments  relating  to  NSTISSP-11  are  included  here  for  completeness  (see  Table 
14).  These  comments  are  not  significant  in  number  and  provide  no  usable  data. 


Table  14.  NSTISSP-11  Expectations  and  Observations 


Stakeholder  Category 

Sequence  Num. 

Description 

Occurrences 

DoD 

Federal  Gov. 

Process 

Producer 

Governance 

Defense  Critical 

Intelligence 

Expectations 

EN11-1 

NTISSP-1 1  should  specify  a  cost  threshold  for  prod¬ 
ucts  requiring  evaluation.  Evaluation  of  inexpensive 
items  is  not  cost  effective. 

2 

“ 

EN11-2 

A  collection  of  protection  profiles  is  needed  before 
NTISSP-1 1  can  be  effectively  applied. 

34 

Observations 

ON  11-1 

NTISSIP-1 1  is  not  well  written. 

34 

Notes:  *  indicates  the  number  of  stakeholder  classes  exceeds  the  number  of  interviews  raising  the  issue.  This  is  due  to  the  fact  that  some 
interviewees  are  placed  in  more  than  one  stakeholder  class. 

♦  issue  identified  in  other  sources  (NIAP  Forum,  Federal  Registry  Announcement,  or  literature 

search) 

5.3.14  Critical  Infrastructure 

An  expectation  expressed  by  stakeholders  from  all  classes  was  that  the  Critical  In¬ 
frastructure  Protection  (CIP)  community  should  be  brought  under  the  national  se¬ 
curity  mandates.  Most  government  departments  and  agencies  that  are  part  of  CIP 
are  already  under  the  FISMA  mandates.  (See  the  discussion  of  policy  issues  in 
Chapter  3.)  However,  including  CIP  under  product  evaluation  and  CC  mandates 
may  create  an  undue  burden  of  cost.  Any  inclusion  of  the  CIP  should  be  deferred 
until  a  more  modestly  priced  evaluation  process  can  be  developed  especially  for 
the  lower  levels  of  assurance  (EAL3  and  below).  However,  many  of  the  products 
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that  have  already  been  evaluated  are  at  assurance  levels  more  suited  to  critical  in¬ 
frastructure  and  the  rest  of  government  than  to  the  national  security  community. 
These  constituencies  should  be  encouraged  to  make  full  use  of  the  advantages  of 
evaluated  products. 


Table  15.  Critical  Infrastructure  Expectations  and  Observations 
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EC  1-1 

The  government  should  impose  information  assur¬ 
ance  requirements  on  the  nation’s  critical  infrastruc¬ 
ture  (banking,  electric,  health,  etc.). 

ECI-2 

The  government  should  not  impose  information  as¬ 
surance  requirements  on  the  nation’s  critical  infra¬ 
structure  (banking,  electric,  health,  etc.). 

ECI-3 

The  concept  of  a  NIAP  advisory  group  representing 
public  (state,  local,  and  commercial)  interests  or  a 
wider  coalition  (partnership  beyond  NSA  and  NIST) 
is  needed 

1 1  ♦ 


OCI-1 


Observations 


Current  laws  and  mandates  are  too  spread  out. 
Need  one  source  for  all. 


24 


iTrm 


Notes:  *  indicates  the  number  of  stakeholder  classes  exceeds  the  number  of  interviews  raising  the  issue.  This  is  due  to  the  fact  that  some 
interviewees  are  placed  in  more  than  one  stakeholder  class. 


♦  issue  identified  in  other  sources  (NIAP  Forum,  Federal  Registry  Announcement,  or  literature 


search) 


5.3.15  Nefarious  and  Malicious  Behavior 

Although  there  was  little  input  on  this  subject,  malicious  code  and  backdoor  ac¬ 
cess  paths  inserted  during  development  have  to  be  considered  in  any  assurance 
arguments.  Many  of  the  interviewees  felt  uncomfortable  discussing  this  area  and 
there  was  little  written  input.  Tools  that  examine  code  and  product  execution  for 
common  security  coding  areas  can  also  examine  the  code  for  some  types  of  these 
activities. 
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Table  16.  Nefarious  and  Malicious  Behavior  Code  Expectations  and  Observations 


Stakeholder  Category 

Sequence  Num. 

Description 

Occurrences 

DoD 

Federal  Gov. 

Process 

Producer 

Governance 

Defense  Critical 

Intelligence 

Expectations 

ENe-1 

Evaluations  should  include  tests  for  mali¬ 
cious  code. 

7* 

ENe-2 

Evaluations  need  not  include  tests  for  mali¬ 
cious  code. 

2 

Notes:  *  indicates  the  number  of  stakeholder  classes  exceeds  the  number  of  interviews  raising  the  issue.  This  is  due  to  the  fact  that 
some  interviewees  are  placed  in  more  than  one  stakeholder  class. 

♦  issue  identified  in  other  sources  (NIAP  Forum,  Federal  Registry  Announcement,  or  literature 

search) 

5.3.16  Comments  Concerning  NIST 

The  last  areas  of  concern  gleaned  from  the  input  materials  concerned  NIST,  and 
they  are  presented  in  Table  17. 


Table  17.  Comments  Concerning  NIST  Expectations  and  Observations 


Stakeholder  Category 

Sequence  Num. 

Description 

Occurrences 

DoD 

Federal  Gov. 

Process 

Producer 

Governance 

Defense  Critical 

Intelligence 

Expectations 

ENi-1 

NIST  involvement  in  NIAP  CCEVS  is  minimal 
and  decreasing. 

7 

ENi-2 

NIST  should  take  a  larger  role  in  the  NIAP/ 
NVLAP  process. 

24 

ENi-3 

NIAP  should  bring  in  additional  partners  beyond 
NSA  and  NIST. 

♦ 
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Notes:  *  indicates  the  number  of  stakeholder  classes  exceeds  the  number  of  interviews  raising  the  issue.  This  is  due  to  the  fact  that  some 
interviewees  are  placed  in  more  than  one  stakeholder  class. 

♦  issue  identified  in  other  sources  (NIAP  Forum,  Federal  Registry  Announcement,  or  literature 
search) _ 


5 . 4  Summary  of  Issues  and  Findings 

The  following  statements  summarize  the  756  recorded  expectations  from  the  collection  of 
interviews,  Forum  discussions,  and  other  contributed  input.  "Issues"  here  represent  the 
principal  concerns  raised  by  interviewees,  Forum  participants,  and  other  contributors. 
"Expectations"  represent  recommendations  expressed  by  these  sources. 

5.4.1  Consumer  Knowledge  and  Understanding 

1.  Issue:  People  with  only  a  basic  familiarity  of  evaluations  often  do  not 
fully  understand  what  an  evaluation  says  about  a  product  and  are  not  able 
to  determine  if  the  product  meets  their  needs. 

Expectation  [ES-01]:  Expanding  the  NIAP  education  programs  for  con¬ 
sumers  would  alleviate  this  problem. 

2.  Issue:  Evaluations  often  contain  confusing  statements  about  what  assur¬ 
ance  aspects  were  evaluated  and  how  the  product  performed. 

Expectation  [ES-02]:  Evaluations  should  state  in  plain  language  what  in¬ 
formation  assurance  protection  the  product  provides. 

5.4.2  Evaluation  Certificates 

3.  Issue:  Evaluation  certificates  contain  little  useful  information. 

Expectation  [ES-03] :  Evaluation  certificates  should  identify  the  degree  of 
security  provided  and  provide  example  applications  for  which  the  product 
is  suitable. 

4.  Issue:  Evaluated  products  are  not  required  to  conform  to  a  well-formed, 
properly  vetted  protection  profile. 

Expectation  [ES-04] :  Conformance  to  a  well-crafted,  properly  vetted  pro¬ 
tection  profile  should  be  made  mandatory. 

5.4.3  Protection  Profiles 

5.  Issue:  Available  protection  profiles  focus  on  special  capabilities  at  higher 
levels  of  assurance. 

Expectation  [ES-05] :  A  collection  of  protection  profiles  covering  core  in¬ 
formation  assurance  capabilities  at  more  modest  assurance  levels  should 
be  developed. 
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5.4.4 


Evaluation  Personnel 


6.  Issue:  There  is  no  requirement  for  evaluators  to  demonstrate  and  maintain 
their  technical  competence. 

Expectation  [ES-06] :  A  credentialing  program  should  be  developed  to  en¬ 
sure  adequate  training  of  evaluators  and  consistent  evaluations  across 
laboratories. 

7.  Issue:  The  conflict  of  interest  rules  governing  evaluation  laboratories  and 
their  personnel  are  weak. 

Expectation  [ES-07] :  The  conflict  of  interest  rules,  particularly  those  for 
developing  evidence  and  conducting  evaluations  on  the  same  products, 
should  be  reviewed  and  strengthened. 

5.4.5  Testing 

8.  Issue:  There  is  no  requirement  for  the  use  of  code  analysis  or  testing  tools 
in  evaluations. 

Expectation  [ES-08]:  The  NIAP  should  develop  and  make  available  a 
standard  collection  of  automated  security  analysis  tools,  and  require  use  of 
these  or  equivalent  tools  in  evaluations. 

9.  Problem:  At  present,  the  Common  Evaluation  Method  (CEM)  and  protec¬ 
tion  profiles  require  no  explicit  vulnerability  testing  in  evaluation  proce¬ 
dures. 

Expectation  [ES-09] :  Both  the  CEM  and  protection  profiles  should  spec¬ 
ify  vulnerability  testing  requirements. 

10.  Issue:  Source  code  review  is  not  required  for  evaluations. 

Expectation  [ES-10] :  Review  of  source  code  should  be  required  at  all 
evaluation  levels.  Developers  may  provide  results  of  automated  tool 
analyses  for  evaluations  at  EAL3  and  below  to  avoid  giving  evaluators  ac¬ 
cess  to  proprietary  code.  See  Annex  F  for  more  information  about  tools. 

5.4.6  Commercial  Viability 

11.  Expectation  [ES-11] :  Market  forces  would  encourage  developers  and  in¬ 
surers  to  warrant  NIAP-evaluated  products  and  assume  at  least  limited  li¬ 
ability  for  information  assurance  breaches. 

Issue:  No  such  product  warranties  have  emerged,  which  implies  that  either 
consumers  do  not  see  added  value  in  warranted,  higher-assurance  prod¬ 
ucts,  or  underwriters  do  not  perceive  evaluations  as  providing  sufficient 
added  assurance,  or  evaluations  are  not  applied  widely  enough  to  develop 
this  market. 

Conjecture  [CES-1]:  Market  forces  would  benefit  by  strengthened 
evaluations  and  wider  application  of  evaluations. 
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5.4.7  Research 

12.  Issue:  A  number  of  open  research  problems  remain  unaddressed,  includ¬ 
ing  assurance  metrics  and  solutions  to  composability  among  other  security 
problems. 

Expectation  [ES-12]:  The  NIAP  should  support  research  in  these  areas. 

5.4.8  Targets  of  Evaluation 

13.  Issue:  The  Target  of  Evaluation  (TOE)  concept  allows  a  developer  to  tai¬ 
lor  what  is  evaluated,  which  is  often  not  the  whole  product  or  is  some  un¬ 
usual  configuration  of  the  product. 

Expectation  [ES-13]:  Whole  products  should  be  evaluated  in  their  normal 
usage  configuration  and  environment. 
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6. 


Areas  of  Concern 


This  Chapter  integrates  the  findings  of  Chapter  3,  Chapter  4,  and  Chapter  5  into  overall 
areas  of  concern  from  which  impacts  and  recommendations  flow.  Chapter  7  describes 
options  for  implementing  these  recommendations.  The  concerns  addressed  here  will  need 
a  phased  approach  to  implementation.  While  tools  can  reduce  costs,  they  must  be  devel¬ 
oped  and  their  utility  demonstrated  before  their  use  can  be  mandated.  The  roadmaps  in 
Chapter  8  address  these  timing  issues  for  each  option. 

6 . 1  Funding  and  Priorities 

Funding  and  priority  shifts  have  moved  the  NIAP  away  from  its  original  intent.  Chapter  3 
found  that  the  lack  of  a  formal  mandate  and  budget  have  limited  the  scope  of  the  NIAP’s 
activities  [FPCy-2],  This,  coupled  with  the  explosive  growth  in  evaluations  documented 
in  Chapter  4  (Section  4.4),  has  caused  the  NIAP  to  focus  almost  exclusively  on  evalua¬ 
tions,  which  are  only  part  of  its  intended  service  [FN-2,  and  FN-3]. 

Observation:  Stretched  NIAP  budgets  and  a  shortage  of  qualified  validators  have  required 
the  NIAP  to  continually  revise  and  rework  its  oversight  of  evaluations. 

Chapter  3  found  that  the  NIAP  is  not  funding  research,  or  tools  as  originally  intended 
[FPCy-3,  FPRe-1,  FPRe-3,  FPRe-4,  FPRe-5],  or  the  derived  requirement  of  education 
and  training  [AN-09,  AN- 17,  EKn-1,  EKn-2,  EKn-3,  EKn-4,  OKn-1,  OKn-2].  Stake¬ 
holders  in  Chapter  5  pointed  out  that  significant  research  gaps  remain  in  infonnation  as¬ 
surance  metrics,  and  in  the  security  of  systems  composed  using  standard  building-block 
components.  Stakeholders  expect  the  NIAP  to  support  research  in  these  areas  [ERe-1, 
ERe-2]. 

Moreover,  as  Chapter  4  found,  budget  constraints  have  precluded  the  NIAP  from  devel¬ 
oping  education  and  training  resources  for  IT  system  consumers,  tools  to  support  secure 
product  development,  and  protection  profiles  for  non-military  applications  [FN-2],  Chap¬ 
ter  5  found  that  stakeholders  expect  the  NIAP  to  develop  and  make  available  a  standard 
collection  of  automated  security  analysis  tools,  and  require  use  of  these  or  equivalent 
tools  in  evaluations  [ETe-1]. 

6 . 2  Product  Evaluation  Focus 

The  NIAP  is  currently  narrowly  focused  on  product  evaluations,  which  is  only  part  of  the 
overall  cybersecurity  landscape.  They  have  actually  performed  well  in  this  limited  scope 
[FN-1,  FN-5].  Chapter  3  documented  that  the  requirement  for  acquisition  of  evaluated  IA 
and  IA-related  products  currently  applies  only  to  the  DoD  and  National  Security  Systems 
[FPAq-1].  No  tie  between  the  DITSCAP  and  evaluated  products  exist.  The  rest  of  the 
federal  government  has  even  less  guidance  on  how  to  choose  IT  security  products  and/or 
integrate  them  with  their  C&A  programs.  Federal  statutes  and  OMB  require  certification 
and  accreditation  (C&A)  of  many  IT  systems,  but  they  do  not  require  any  interaction  be- 
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tween  the  NIAP  product  evaluations  and  C&A  activities  [FPAq-2].  C&A  is  a  required  ac¬ 
tivity  and  will  be  perfonned  with  or  without  product  evaluation.  Currently,  the  results  of 
a  product  evaluation  (evidence,  documentation)  are  not  specifically  intended  for  system 
security  certification  agents — they  are  primarily  developed  for  those  involved  in  the 
evaluation/validation  process.  To  the  extent  that  the  NIAP/CCEVS  can/does  change  their 
documentation  &  evidence  to  also  meet  the  needs  of  system  security  certification  agents 
(and  system  auditors,  system  developers/integrators),  these  evaluation  deliverables  will 
be  more  useful.  In  Chapter  5  stakeholders  were  concerned  that  the  details  of  analysis  and 
testing  results  contained  in  the  NIAP  evaluation  technical  reports  are  not  available  to 
DITSCAP  and  FISMA  C&A  processes  [ECa-2,  ECa-3,  ECa-5,  OCa-1], 

6 . 3  Cybersecurity  Changes  Since  the  NIAP  Establishment 

The  cybersecurity  landscape  has  shifted  while  the  NIAP  has  struggled  to  keep  up  with 
evaluations.  The  finding  that  the  growth  of  evaluations  has  taxed  the  pool  of  validators 
[FN-3]  is  exacerbated  by  the  complexity  and  confusion  of  cybersecurity  policies  and 
standards  [FPCy-l].In  many  cases  the  technology  is  changing  faster  than  formal  stan¬ 
dards  processes  are  able  to  track.  This  makes  it  difficult  for  government  agencies  to  de¬ 
termine  which  standards  to  use  [FPSt-1].  Moreover,  policies  evolve  so  rapidly,  imposing 
new  requirements  and  superseding  each  other,  that  government  procurement  officials  of¬ 
ten  cannot  determine  which  requirements  apply  to  their  particular  situation  [FPCy-1]. 

In  addition,  the  Common  Criteria  upon  which  the  NIAP  evaluations  are  based  has  not 
kept  up  with  changes  in  the  cybersecurity  landscape.  For  example,  stakeholders  expect 
more  rigorous  testing  and  automated  source  code  review  at  all  levels  of  assurance  [FETe- 
1,  FETe-2,  FETe-3,  ES-08,  ES-09,  ES-10],  Other  necessary  changes  to  the  CC  and 
CCEVS  process  that  were  identified  include: 

•  Requiring  evaluators  to  review  software  source  code  in  depth  for  security 
purposes.  This  review  must  be  automated  due  to  shear  volume  and  com¬ 
plexity  [FETe-3].  In  the  CCEVS,  source  code  is  not  even  available  at  the 
lower  evaluation  assurance  levels. 

•  Requiring  the  use  of  automated  tools  to  identify  critical  code  components 
is  pennitted  but  not  required  at  any  level. 

6 . 4  Continuing  Cyberspace  Changes 

Cyberspace  continues  to  evolve  technologically  and  globally.  As  a  result,  advanced  tech¬ 
niques  such  as  grid  computing,  distributed  intelligent  agent  systems,  distributed  knowl¬ 
edge  management,  and  large-scale  distributed  systems  of  systems  are  being  built  without 
adequate  protection  mechanisms  and  will  be  susceptible  to  cyber  attack.  Additionally, 
within  a  few  years,  the  largest  makers  of  computer  chips,  the  largest  integrators  of  com¬ 
puters  and  network  devices,  and  the  largest  producers  of  software  will  all  be  foreign 
sources.  This  raises  the  level  of  anxiety,  if  not  the  actual  risk,  of  products  containing  hid¬ 
den  malicious  functionality.  CC  evaluations,  at  present,  do  not  provide  sufficient  screen¬ 
ing  to  detect  potential  product  behavior  that  is  inconsistent  with  the  product’s  intended 
use  for  high  assurance  systems  [ENe-1]. 


82 


6 . 5  Common  Criteria  Evaluation  Costs 

Common  Criteria  evaluations  cost  too  much.  This  is  especially  true  for  low  assurance 
products.  Chapter  5  found  that  the  high  costs  of  evaluation  often  inhibit  the  use  of  appro¬ 
priate  small  business  and  open  source  software  [OCT-1,  OCT-3],  The  cost  of  evaluations 
is  assumed  to  be  insignificant  (or  at  least  not  prohibitive)  so  that  small  businesses  and 
independent  developers  can  have  products  evaluated  as  well  as  large  corporations.  How¬ 
ever,  an  evaluation  may  cost  as  little  as  $30,000  to  as  much  as  $1,000,000  or  more  de¬ 
pending  on  a  number  of  issues.  This  cost  level  for  an  evaluation  may  be  a  barrier  to  entry 
for  open  source  software  (OSS)  and  many  products  developed  by  small  businesses  [OCT- 
3].  As  a  result,  many  products  may  not  be  evaluated.  By  requiring  use  of  evaluated 
products,  government  policies  eliminate  products  that  may  be  more  appropriate  and  suffi¬ 
ciently  secure  for  the  task.  Small  business  stakeholders  interviewed  in  Chapter  5  stated 
that  evaluation  costs  are  too  high  and  they  take  too  long  [ECT-3,  OCT-1,  OCT-2],  Sev¬ 
eral  interviewees  believed  that  alternative  assurance  methods  are  needed,  especially  to 
reduce  costs  (such  as  a  “CC  life”)  [EAa-3],  Annex  H  discusses  the  alternate  assurance 
aspects  in  some  detail. 

Relaxation  of  some  of  the  documentation  requirements  at  all  levels  is  recommended. 
Documentation  is  expensive;  the  labs  can  detennine  when  they  have  enough  infonnation 
of  the  right  type. 

6 . 6  Policy  and  Legal  Landscape 

Chapter  3  found  that  the  cybersecurity  policy  landscape  is  complex  and  confusing.  Poli¬ 
cies  are  rapidly  evolving,  with  both  legislation  and  department  and  agency  policies  im¬ 
posing  new  requirements  and  superseding  each  other.  There  is  no  single  source  for  up¬ 
dated  current  policies,  which  makes  it  difficult  for  government  procurement  offices  to 
determine  which  requirements  apply  to  their  particular  situation  [FPCy-1]. 

6 . 7  Education,  Training,  and  Awareness 

Education,  Training,  and  Awareness  Programs  have  languished,  are  incomplete,  and  not 
current  [FPEta-1,  FPEta-2],  There  is  a  critical  need  for  education  and  training  on  several 
levels.  IT  product  developers  need  a  trained  workforce  to  develop  secure  products. 
Chapter  4  documents  that  developers  are  assumed  to  know  about  common  mistakes,  how 
they  lead  to  security  vulnerabilities,  and  how  to  avoid  those  mistakes  [AN-09]. 

Stakeholders  in  Chapter  5  expect  the  NIAP  to  require  evaluators  to  demonstrate  and 
maintain  their  technical  competence  [EPe-2,  EPe-3].  In  addition,  they  expect  the  NIAP 
to  impose  tighter  constraints  on  evaluation  labs  developing  evidence  for  products  they 
had  in  evaluation  [EPe-1,  EPe-4]. 

Finally,  government  IT  system  acquisition  offices  and  system  operators  need  a  trained 
workforce  to  buy,  implement,  and  maintain  secure  IT  systems.  As  Chapter  5  found,  con¬ 
sumers  with  only  a  basic  familiarity  of  the  NIAP  processes  often  do  not  fully  understand 
what  an  evaluation  says  about  a  product  and  are  not  able  to  detennine  if  a  product  meets 
their  needs  [EKn-3,  OKn-1,  OKn-2]. 
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6 . 8  Flexible  and  Capably  Staffed  Program 

Chapter  4  determined  that  NSA  and  NIST,  without  separate  funding  earmarked  for  the 
NIAP,  have  produced  a  flexible,  capably  staffed,  although  under-funded  product  evalua¬ 
tion  system  [FN-1].  Moreover,  it  concluded  that  the  shortcomings  identified  for  the  NIAP 
are  addressable  without  a  radical  overhaul  of  the  NIAP’s  product  evaluation  scheme.  Re¬ 
vamping  the  administrative  structure,  processes,  and  expertise  needed  to  create  an  alter¬ 
nate  product  evaluation  scheme  would  be  costly  and  time  consuming,  and  there  is  little 
evidence  that  a  more  successful  result  would  be  achieved  [RN-1]. 

6 . 9  Return  on  Investment 

There  is  no  ROI  calculation  for  evaluated  products.  As  Chapter  3  points  out,  the  memo¬ 
randum  establishing  the  NIAP  calls  for  research  to  develop  objective  measures  of  quality 
and  security,  but  because  of  funding  limitations,  this  research  has  not  been  conducted 
[FPRe-4].  A  particular  area  of  research  that  is  being  neglected  is  that  of  metrics  to  help 
consumers  determine  the  return  on  investment  of  evaluation.  Stakeholders  in  Chapter  5 
reiterated  that  metrics  that  both  quantify  security  aspects  of  software  and  the  effective¬ 
ness  of  evaluations  are  sorely  lacking  [ERe-1,  ERe-2,  ORe-1]. 

6.10  Maintenance  Assurance  and  Flaw  remediation 

CC  evaluations  have  not  adapted  to  the  development  paradigm  of  release,  patch,  and  up¬ 
date.  Software  producers  often  create  new  releases  of  products  with  defect  corrections 
and  minor  functional  enhancements.  At  present,  the  standard  CC  assurance  packages  for 
EAL1  through  7  have  no  provisions  for  accommodating  such  minor  changes.  While 
maintenance  assurance  and  flaw  remediation  packages  are  part  of  the  standard,  they  are 
not  part  of  any  assurance  packages.  Deviations  from  the  standard  assurance  packages 
are  seldom  made  (a  notable  exception  is  that  several  NSA  developed  PPs  contain  some  of 
these  provisions).  Each  new  release  may  therefore  be  treated  as  a  new  product,  requiring 
full  evaluation.  A  number  of  stakeholders  felt  that  analysis  and  testing  of  minor  product 
changes,  which  would  require  considerably  less  effort  than  a  full  evaluation,  should  be 
sufficient  to  extend  the  original  certificate  to  cover  the  updated  product  [EAM-1].  These 
provisions  occur  within  the  maintenance  assurance  packages.  Further,  it  is  expected  that 
where  security  issues  are  involved,  security  failures  and  the  patches  to  improve  the  prod¬ 
uct  should  be  accompanied  by  notification  to  all  registered  users  [EAM-2],  These  provi¬ 
sions  occur  in  the  flaw  remediation  packages. 

6.11  Evaluation  Assurance 

Evaluation  assurance  in  the  documented  packages  concentrates  on  documentation.  The 
standard  does  allow  for  alternate  assurance  methods.  The  NIAP  CCEVS  as  documented 
in  Chapter  4  requires  no  specific  vulnerability  analysis  or  testing  methods  that  are  releas¬ 
able  [FN-6].  Commercial  software  development  practices  differ  widely  and  produce  dif¬ 
ferent  documentation,  as  well  as  products  with  different  types  of  defects.  Requirements 
tracing  through  design  and  coding  is  assumed  to  be  an  effective  security  analysis  tech¬ 
nique,  although  there  is  little  evidence  that  this  contributes  significantly  to  exposing  secu¬ 
rity  flaws. 
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Stakeholders  interviewed  in  Chapter  5  agreed  that  the  documentation  required  for  evalua¬ 
tions  is  partly  responsible  for  the  high  costs  of  evaluations  [OCT-4].  While  it  is  true  that 
the  CC  does  not  require  specific  fonns  of  documentation,  it  does  spell  out  content  re¬ 
quired  by  assurance  elements.  Little  of  the  documentation  presented,  as  evidence  is  part 
of  nonnal  development  processes.  This  documentation  is  expensive  to  produce  and  often 
requires  revision  to  satisfy  evaluators. 

6.12  Nefarious  and  Malicious  Code 

Evaluations  ignore  the  possibility  of  nefarious  and  malicious  code.  Chapter  4  found  that 
product  developers  are  assumed  to  be  trustworthy  and  would  not  knowingly  insert  mali¬ 
cious  code  into  their  products  [AN- 18,  AN- 19].  Stakeholders  (Chapter  5)  believe  that  ma¬ 
licious  code  and  backdoor  access  paths  inserted  during  development  have  to  be  consid¬ 
ered  in  any  assurance  arguments  [ENe-1]. 

Chapter  3  found  that  while  sources  such  as  the  2001  Report  of  the  Defense  Science  Board 
on  Defensive  Information  Operations  note  the  need  for  research  in  scalable  global  com¬ 
puting,  mobile  code  security,  fault  tolerance,  and  malicious  code  detection  they  do  not 
call  for  research  to  improve  the  CCEVS  or  the  NIAP  or  for  research  in  technologies  di¬ 
rectly  related  to  them. 

While  it  is  realized  that  the  detection  of  an  arbitrary  piece  of  malicious  code  is  not  cur¬ 
rently  solvable,  tools  can  be  developed  to  look  for,  and  discover  certain  types  of  more 
common  malicious  code.  This  would  be  of  general  benefit. 

6.13  Common  Criteria  Issues 

Several  of  the  preceding  sections  dealt  with  adding  work  to  evaluations  when  doing 
Common  Criteria  evaluations.  Three  areas  in  particular  are  worth  further  discussion: 
vulnerability  testing,  flaw  remediation,  and  maintenance  of  assurance.  All  three  are  actu¬ 
ally  in  the  Common  Criteria,  but  not  explicitly  called  out  in  assurance  packages  or  not  at 
a  high  enough  level  (in  the  case  of  vulnerability  work).  Table  18  shows  the  Common  Cri¬ 
teria  the  assurance  as  taken  from  Part  3  of  the  CC  standard.  Assurance  Maintenance, 
which  is  a  stand-alone  assurance  class  in  the  standard,  has  been  added  to  the  table.  It 
should  be  specifically  noted  that  Flaw  Remediation  (AFCFLR)  and  Maintenance  of  As¬ 
surance  packages  (labeled  as  the  Maintenance  Assurance  class  in  the  table),  are  not  in¬ 
cluded  in  any  current  assurance  packages.  Further,  the  stronger  parts  of  the  vulnerability 
analysis  (AVA_VFA.3“  and  above)  occur  only  at  levels  above  those  covered  by  the 
MRA.  The  latter  is  the  subject  of  DoD  protection  profile  additions  to  standard  packages. 
The  analysis  team  felt  that  these  three  should  be  included  in  all  evaluations  of  IA  and  IA 
enable  software. 


23  AVA_VLA3  is  the  3rd  level  of  vulnerability  analysis  which  in  the  table  is  not  required  until  EAL5. 
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Table  18.  Evaluation  Assurance  Level  Summary 


MRA 


Assurance 

Class 

Assurance 

Family 

Assurance  Components  by  Evaluat 

on  Assurance  Level 

EAL1  EAL2 

EAL3 

EAL4 

EAL5 

EAL6 

EAL7 

Configuration 

management 

ACM  AUT 

1 

1 

2 

2 

ACM  CAP 

1  2 

3 

4 

4 

5 

5 

ACM  SCP 

1 

2 

3 

3 

3 

Delivery  and 
operation 

ADO  DEL 

1 

2 

2 

2 

3 

3 

ADO  IGS 

i  i 

1 

1 

1 

1 

1 

Development 

ADV  FSP 

i  i 

1 

2 

3 

3 

4 

ADV  HLD 

1 

2 

2 

3 

4 

5 

ADV  IMP 

1 

2 

3 

3 

ADV  INT 

1 

2 

3 

ADV  LLD 

1 

1 

2 

2 

ADV  RCR 

i  i  i 

1 

2 

2 

3 

ADV  SPM 

1 

3 

3 

3 

Guidance 

documents 

AGD  ADM 

1  1 

1 

1 

1 

1 

1 

AGD  I JSR 

i  i 

1 

1 

1 

1 

1 

Life  cycle 

AT.C  DVS 

1 

1 

1 

2 

2 

Ai  r  FI  R 

ALC  LCD 

1 

2 

2 

3 

support 

AT.C  TAT 

1 

2 

3 

3 

Maintenance 

Assurance 

AMA  AMP 

AMA  CAT 

AMA  EVD 

ALC  SIA 

Tests 

ATE  COV 

1 

2 

2 

2 

3 

3 

ATE  DPT 

1 

1 

2 

2 

3 

ATE  FUN 

1 

1 

1 

1 

2 

2 

ATE  IND 

1  2 

2 

2 

2 

2 

3 

Vulnerability 

assessment 

AVA  CCA 

1 

2 

2 

AVA  MSU 

1 

2 

2 

3 

3 

AVA  SOF 

1 

1 

1 

1 

1 

1 

AVA  VI  A 

_ 1 _ 

_ i _ 

_ 2__ 

_ 1 

4 

4 

*  m*  ■ - 11 - — 

MRA  ^ 

Beginning  of  serious  analysis 


6.14  Targets  of  Evaluation 

The  TOE  concept  allows  a  developer  to  tailor  the  product  that  is  evaluated,  which  is  often 
not  the  whole  product  or  is  some  limited  configuration  of  the  product.  While  this  is  meant 
to  tailor  the  evaluation  to  the  IA  aspects  of  the  product  and  not  cover  the  non-IA  func¬ 
tionality,  it  is  sometimes  used  to  remove  IA  aspects  (listed  as  optional,  but  not  in  the 
evaluated  configuration)  from  evaluation.  This  is  somewhat  confusing  to  the  consum¬ 
ers/acquirers  of  a  product  [ETOE-1].  Any  feature  of  an  evaluated  product  that  can  be 
accessed  by  the  consumer/user  should  be  part  of  the  evaluation.  Stakeholders  expect 
whole  products  to  be  evaluated  in  their  normal  usage  configuration  and  environment. 

6.15  Conflicts  and  Compromise 

It  is  clear  that  all  expectations  cannot  be  met.  In  fact  there  are  several  of  the  findings  that 
conflict  with  expectations  as  well  as  expectations  that  conflict  with  expectations.  Some 
examples: 
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6.15.1  Intellectual  Property  and  the  need-to-know 

Market  forces  that  are  discussed  next,  often  dictate  that  a  product  may  be  differentiated 
by  its  unique  intellectual  property.  For  this  reason  controls  are  put  in  place  to  protect 
those.  There  is  a  basic  conflict  between  providing  source  code  for  analysis  and  the  pro¬ 
tection  of  that  property.  One  compromise  might  be  a  standard  set  of  tools  and  self  certifi¬ 
cation  at  low  levels  of  assurance.  Another  conflict  is  the  evaluation  laboratory  processes 
that  need  protection,  and  the  need  for  system  certifiers  to  see  exactly  what  has  been  tested 
and  the  results  of  those  tests.  This  has  lead  to  evaluation  technical  reports  being  unavail¬ 
able.  A  compromise  might  be  sanitized  reports  where  applicable.  Finally,  the  whole 
problem  of  metrics,  tracking  and  return-on-investment  is  complicated  by  lab  proprietary 
cost  methodology.  Agreements  should  be  worked  out  where  this  data  is  made  available 
for  use  in  analyses,  while  maintaining  the  confidentiality  of  the  data. 

6.15.2  Market  forces 

The  expectation  for  clear  market  viability  [EMR-3]  and  some  relation  to  reliability  issues 
[OMR-1],  leads  to  a  conflict.  Product  evaluation  will  be  sought  after  without  mandate,  if 
it  is  in  the  best  interest  of  the  vendor.  Market  forces  can  bring  this  about  by  reducing 
vendor  costs  or  exposures  (liability  issues)  or  product  differentiation  either  increasing 
market  share  or  product  worth.  The  consumer  can  only  provide  the  market  worth  vari¬ 
able.  Here  Education,  Training  and  Awareness  are  key.  Consumers  may  refuse  to  buy  a 
lamp  for  their  home  if  it  is  not  UL  listed  and  therefore  UL  listing  has  worth  to  the  pro¬ 
ducer.  The  cost,  however,  (discussed  next)  must  be  small  enough  that  the  increased  mar¬ 
ket  share  or  product  worth  can  recover  the  expenses.  Consumers  must  know  what  an 
evaluated  product  brings  to  the  table,  and  this  has  not  been  adequately  articulated.  On  the 
other  hand,  mandated  evaluation,  when  favorable  will  be  incorporated  into  vendors  mar¬ 
keting.  An  example  would  be  a  high  EPA  mileage  rating  which  will  be  touted  as  a  prod¬ 
uct  differentiator.  The  government  must  let  the  market  do  what  it  can,  and  encourage 
market  solutions.  The  market,  however,  sometimes  needs  help.  When  and  where  man¬ 
dates  are  needed,  and  under  what  circumstances  are  sophisticated  and  subtle  trade-offs. 

6.15.3  Cost  of  Evaluation 

The  clear  expectation  is  that  evaluation  costs  should  be  reduced  [OCT-1,  ECT-1,  ECT-2]. 
At  the  same  time,  it  is  also  clear  that  stakeholders  expect  a  more  meaningful,  technically 
detailed,  evaluation  [ECm-1,  EPP-2,  ETe-2,  ETe-3,  etc.].  These  latter  sets  of  expectations 
drive  the  cost  up.  The  compromise  is  to  provide  enough  assurance  in  the  cybersecurity 
space  to  meet  needs  without  adding  unnecessary  costs.  There  appears  to  be  a  split  in  the 
degree  of  assurance  needed.  It  is  clear  that  for  national  security  systems,  defense  critical 
applications,  some  banking  transactions,  and  others  that  a  high  level  of  assurance  is 
needed.  In  these  systems  the  potential  for  loss  is  extremely  high  and  probably  worth  the 
cost  of  a  very  thorough  evaluation.  Other  systems,  (such  as  home  computer  use,  protect¬ 
ing  fish  and  wildlife  parks  data,  etc.)  require  pitfalls  and  poor  programming  techniques 
that  lead  to  undue  vulnerabilities  are  avoided.  There  exists  a  natural  split  between  the 
low  assurance  and  high  assurance  products. 

On  the  low  assurance  end,  it  is  suggested  that  the  vendor  himself  can  be  certified  through 
a  modification  of  processes  like  the  SEI  CMM.  Under  this  approach,  an  annual  or  bien- 
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nial  evaluation  would  look  at  quality  control,  configuration  control,  flaw  remediation, 
Security  processes  and  maintenance  assurance  processes.  This  look  may  cost  $20-30,000 
and  can  be  done  in  one  week.  With  this  certificate  in  place  insuring  a  product  was  pro¬ 
duced  using  the  system,  faithfully  provides  advertised  functionalities  (functional  testing), 
and  is  relatively  free  from  bad  programming  and  known  commonly  exploited  vulnerabili¬ 
ties  (tools  can  help  here).  The  emphasis  would  be  on  testing  not  documentation.  Re¬ 
markably,  this  can  be  done  under  the  Common  Criteria,  which  allows  the  PP  and  ST  to 
specify  its  own  assurance  verification  processes.  It  is  not  clear  that  the  Common  Criteria 
Recognition  Agreement  would  provide  mutual  recognition  without  some  standardization 
of  which  and  how. 

On  the  high  assurance  end,  the  full  Common  Criteria  approach  is  needed  with  increased 
vulnerability  testing  (tools  can  help  here),  flaw  remediation  and  maintenance  assurance 
packages  added. 

6.15.4  Compromise 

While  there  are  conflicts  in  needs,  expectations,  and  implementation  requirements,  there 
are  compromises  that  stem  from  the  goals  being  pursued.  The  next  two  chapters  are 
dedicated  to  reviewing  these  options  and  providing  a  roadmap  of  phased  in  compromise 
actions  to  provide  for  the  general  raising  of  cybersecurity  awareness  and  overall  system 
and  product  security. 
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7. 


Options 


7 . 1  Introduction 

Although  a  number  of  findings  and  recommendations  have  been  presented  in  the  previous 
chapters,  simply  addressing  the  individual  problems  illustrated  will  not  solve  the  over¬ 
arching  concerns  presented  in  Chapter  6.  Moreover,  two  shifts  of  changes  in  the  external 
environment  have  occurred.  The  first  is  a  shift  from  government  ownership  and  control 
of  data  to  government  ownership  of  data  on  civilian  systems.  The  second  is  a  shift  away 
from  government-owned  systems  upon  which  government  decisions  are  based.  There  are 
seven  environmental  changes  of  note: 

(1)  Change  in  systems  ownership; 

(2)  Change  in  data  storage  locations; 

(3)  Change  in  computing  and  network  capacity; 

(4)  Change  in  user  expertise; 

(5)  Change  to  large,  inter-connected  government  decision  support  systems; 

(6)  Change  in  balance  of  static  versus  dynamic  system  composition  toward  dynamic 
system  composition,  and  finally, 

(7)  Change  in  user  demands  and  expectations  of  those  systems. 

Based  on  an  analysis  of  the  findings,  recommendations  and  areas  of  concern,  IDA  has 
identified  six  options  for  the  NIAP,  in  increasing  magnitude  of  change.  The  first  three 
options  do  not  respond  to  the  changes  in  the  external  environment,  but  present  internal 
organizational  changes  to  “improve  the  process”.  The  next  two  options  (Options  4  and  5) 
acknowledge  that  the  environment  external  to  the  NIAP  has  changed  significantly,  and 
describe  changes  in  the  NIAP  in  response  to  that  external  environment  along  with  com¬ 
plementary  internal  changes.  The  final  option  (Option  6)  not  only  acknowledges  the 
changes  in  the  external  environment,  but  also  assumes  that  incremental  changes  in  the 
process  are  an  insufficient  response  and  that  a  radical  change  in  thinking,  processes  and 
organization  (or  “transformation”)  is  required. 

Options  3,  4  and  5  build  upon  their  predecessors  and  contain  examples  of  the  types  of 
issues  addressed  by  the  option  with  the  arguments  to  be  made  both  for  and  against.  The 
set  of  activities  within  each  option  is  not  complete,  but  is  intended  to  give  a  general  feel 
for  the  overall  focus  of  the  option.  Decision-makers  may  choose  to  implement  only  parts 
of  the  options,  or  one  option  plus  portions  of  another.  A  roadmap  for  each  of  the  options 
is  presented  in  Chapter  8. 

The  final  option  could  be  executed  in  parallel  with  any  one  of  the  Options  1  through  5. 
The  options  are  as  follows: 

Option  1:  Eliminate  the  NIAP 
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Option  2:  Continue  the  NIAP  in  its  current  form  (reduced  from  the  original  intent) 

Option  3:  Restore  the  NIAP  to  the  original  intent  of  the  Letter  of  Partnership  between 
NS  A  and  NIST 

Option  4:  Modernized  approach  to  cybersecurity,  but  update  the  partnership  to  reflect 
changes  in  the  environment  since  its  creation  in  1998 

Option  5:  Integrated  approach  to  cybersecurity 

Option  6:  Forward  looking  approach  to  cybersecurity  (new  paradigm) 

7 . 2  Descriptions  of  Options 

The  options  below,  with  the  exception  of  Option  1,  are  addressable  within  the  current 
structure  of  the  NIAP  Cost  estimation  was  not  done  because  of  the  large  number  of  dif¬ 
ferent  implementations  that  can  be  undertaken.  However  Rough  Order  of  Magnitude 
(ROM)  values  are  as  follows;  Option  2  is  double24,  Option  3  is  four  times,  Option  4  is  six 
times  and  Option  5  is  eight  times  the  current  funding.  Option  6  does  not  stand  alone,  and 
is  totally  dependant  upon  which  elements  of  its  companion  option  are  pushed  into  Option 
6. 

7.2.1  Option  1:  Eliminate  the  NIAP 

Description:  With  this  option,  the  emphasis  on  product  evaluation  is  shifted  to  C&A  of 
systems.  Separate  efforts  on  software  assurance  are  addressing  the  development  of  more 
secure  software  and  applications  from  inception.  This  front-end  effort  would  then  become 
far  more  important,  with  the  assumption  that,  if  software  and  applications  are  developed 
in  a  more  secure  fashion,  there  would  be  a  significantly  reduced  need  to  do  evaluation  of 
completed  products.  For  those  exceptional  cases  where  additional  assurance  is  needed, 
NS  A  would  accomplish  the  evaluations  required  (as  they  do  now),  but  the  need  for  what 
are  currently  the  EAL4  and  below  product  evaluations  would  be  removed.  NIST  would 
still  continue  to  produce  standards  for  the  Federal  community,  as  directed  by  FISMA,  fo¬ 
cusing  on  the  C&A  process  and  product  and  system  configuration  guidelines  to  ensure 
not  only  secure  composition  into  a  system,  but  also  secure  operation  within  a  Federal  en¬ 
tity’s  IT  infrastructure. 

There  are  four  versions  of  this  option,  all  with  different  emphases: 

1)  Keep  CCEVS,  NSA  takes  the  lead  for  DoD/NSS 

2)  Eliminate  product  evaluation,  drop  out  of  CCEVS,  focus  on  improving  C&A 

3)  Keep  product  evaluation,  but  do  it  another  way  (alternative  fonns  of  evaluation) 

4)  Create  a  hybrid  of  versions  2  and  3,  only  specific  evaluations  done  an  alternate  way 
of  assurance  or  CC. 


24  To  continue  the  NIAP  in  its  current  form,  the  funding  mist  be  increased  to  match  the  increase  in  evalua¬ 
tion  workload. 
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Pros:  This  is  a  relatively  simple  option  to  implement,  since  the  requirements  for  C&A 
exist  across  the  Federal  government  and  NIST  is  already  addressing  standards  for  C&A 
and  configuration  guidelines.  Where  product  evaluation  may  still  be  required  for  those 
systems  of  highest  sensitivity,  NS  A  has  the  expertise  and  processes  to  do  so  for  those  ac¬ 
tivities  (such  as  NSS  and  IC)  requiring  evaluations,  and  alternatives  to  the  CCEVS  are 
available  to  other  Federal  entities  that  wish  to  use  them.  This  could  also  reduce  the  dupli¬ 
cation  of  testing  efforts  between  C&A  and  product  evaluation  and  focus  the  efforts  on 
C&A. 

Cons:  This  option  does  not  recognize  any  changes  in  the  cybersecurity  environment.  It 
would  most  likely  disqualify  us  from  the  MRA.  DoD  and  NSS  are  still  required  to  use 
evaluated  products  unless  DoD  and  NS  A  are  willing  to  change  this  requirement.  It  is  un¬ 
known  (has  not  been  evaluated)  whether  the  alternative  evaluation  methods  are  sufficient 
to  address  this  need.  This  option  puts  significant  additional  emphasis  on  development  of 
software  assurance  processes  and  methodologies  that  currently  are  not  mature.  It  also 
puts  the  primary  burden  of  cybersecurity  on  C&A  processes,  and  assumes  that  accreditors 
are  knowledgeable  enough  and  the  processes  are  rigorous  enough  to  ensure  that  the  proc¬ 
ess  results  in  a  secure  system  ready  to  operate.  If  the  version  of  this  option  is  to  not  do 
product  evaluations  at  all,  it  could  be  seen  as  turning  our  back  on  evaluations  and  stating 
that  the  labs  are  not  commercially  viable  -  a  loss  of  investment  on  the  part  of  the  labs. 

7.2.2  Option  2:  Continue  the  NIAP  in  its  Current  Form 

Description:  In  this  option,  the  NIAP  would  continue  to  be  a  partnership  between  NIST 
and  NSA,  but  it  would  continue  to  almost  exclusively  monitor  product  evaluations  and 
interact  with  the  Mutual  Recognition  Arrangement  partners  on  improvements  to  the 
Common  Criteria.  More  personnel  would  be  needed  to  handle  the  growth  of  evaluations. 
The  complex  legal  and  guidance  issues  could  be  clarified  by  the  use  of  a  legal  clearing¬ 
house  (see  chapter  3  for  a  discussion  of  statutory  and  policy  requirements).  This  clearing¬ 
house  is  clearly  over  and  above  the  previous  NIAP  functionality,  as  are  many  others  we 
will  suggest. 

Pros:  As  this  is  a  continuation  of  the  current  state,  there  are  no  obstacles  to  implement, 
so  this  option  presents  the  lowest  technical  risk.  Any  other  needs  will  have  to  be  met  by 
the  development  of  new  processes/practices  outside  of  the  NIAP/CCEVS. 

Cons:  Budget  issues  would  have  to  be  resolved  through  the  agencies.  Like  Option  1, 
this  Option  ignores  or  does  not  respond  to  the  significant  changes  in  the  cybersecurity 
environment.  What  the  NIAP  does  and  how  the  responsibilities  are  assigned  within  the 
partnership  depends  on  the  current  budgets  and  spending  priorities  of  partners.  Without 
making  significant  interfaces  to  C&A,  much  of  what  can  be  accomplished  with  product 
evaluation  may  not  be  reused  and  in  that  sense,  can  be  viewed  as  a  duplicative  or  waste¬ 
ful  effort.  Leaving  the  NIAP  in  its  current  reduced  state  of  operations  only  satisfies  the 
formality  of  having  an  evaluation  process  to  replace  the  Orange  Book  (CSC-STD-001-83, 
Trusted  Computer  System  Evaluation  Criteria,  National  Computer  Security  Center 
(NCSC))  evaluations  of  the  past.  To  continue  the  NIAP  in  the  current  form  the  funding 
must  be  increased  to  match  the  increase  in  evaluation  workload. 
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7.2.3  Option  3:  Restore  the  NIAP  (to  the  original  intent  of  the  Letter  of  Part¬ 
nership  between  NSA  and  NIST) 

Description:  This  option  restores  the  NIAP  to  the  full  functioning  envisioned  when  the 
partnership  was  first  established  in  1998,  without  regard  to  the  changes  in  the  environ¬ 
ment  that  have  occurred  since  its  inception.  The  partnership  between  NIST  and  NSA 
would  be  formalized  in  some  way  (more  formally  than  the  existing  Letter  of  Partnership 
between  the  parties),  with  the  responsibilities  of  each  party  spelled  out  in  detail:  i.e., 
NIST  certifies  evaluation  laboratories,  provides  a  full-time  person  to  work  with  commer¬ 
cial  sectors  (finance,  health  care,  telecommunications,  etc.)  to  specify  their  security  re¬ 
quirements,  and  participates  in  the  CC.  NSA  would  be  required  to  provide  enough  vali¬ 
dators  to  adequately  oversee  validations  done  by  the  labs,  handle  validations  above 
EAL4,  and  participate  in  the  CC.  Funding  will  be  a  real  issue,  as  NIST  and  NSA  have 
not  agreed  to  fund  the  NIAP  at  these  levels  in  the  past. 

The  original  vision  was  that  the  NIAP  would  produce  tools,  provide  education  about  the 
CC,  and  write  security  specifications.  Also  part  of  its  original  scope  (see  Section  1.2)  was 
to  “Foster  research  and  development  in  security  tests  methods,  and  metrics”'  .  Part  of 
the  original  scope  was  the  development  of  PPs  beyond  the  DoD  to  other  government  de¬ 
partments,  critical  infrastructure,  and  private  sectors  as  the  tasking  has  or  shOould  evolve 
to  provide  education  to  include  writing  PPs  and  STs  and  interpreting  evaluation  certifi¬ 
cates.  These  functions  would  now  be  performed  as  originally  intended. 

Pros:  The  primary  focus  of  this  option  is  the  restoration  of  the  NIAP  to  its  original  mis¬ 
sion  and  a  fonnal  recognition  of  this  mission.  This  option  would  clarify  duties  and  re¬ 
sponsibilities  of  the  parent  organizations,  set  out  detailed  requirements  and  provide  for 
oversight  mechanisms.  The  analysis  tools  would  lead  to  better  testing  at  a  lower  cost. 
The  development  of  tools  for  automating  the  simpler  aspects  of  security  evaluation  could 
raise  the  security  level  of  all  evaluated  products.  The  NIAP  would  also  perform  other 
critical  functions  required  to  make  product  evaluation  both  understandable  and  useful,  as 
well  as  the  product  evaluation  core  function. 

Cons:  Budget  requirements  would  have  to  be  resolved  through  the  agencies,  and  they 
have  not  yet  been  able  to  agree  to  full  funding  for  this  option  except  in  the  beginning 
shortly  after  the  MOU  was  signed.  Additional  funding  may  have  to  be  provided  to  NIST 
and  NSA.  As  with  Options  1  and  2,  this  option  does  not  address  any  changes  indicated 
by  the  current  environment.  This  option  also  does  not  address  the  issue  of  how  the  prod¬ 
uct  evaluation  fits  into  the  C&A  process  which  makes  little  use  of  product  evaluations 
and  duplicates  some  of  the  effort  and  cost  of  the  product  evaluations,  especially  where 
these  product  characteristics  implement  system-level  controls.  This  option  also  does  not 
address  how  product  evaluations  fit  in  the  overall  Federal  cybersecurity  effort.  Because  it 
does  not  address  some  of  the  fundamental  concerns  about  the  product  evaluation  process, 
although  this  may  be  a  desirable  option  to  address  short-tenn  issues,  in  the  long  run,  it 
may  not  be  a  justifiable  expenditure  of  scarce  resources. 


25  Letter  of  Partnership  National  Security  Agency  and  National  Institute  of  Standards  and  Technology,  Au¬ 
gust  22,  1997.  (See  Annex  G). 
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7.2.4  Option  4:  Modernistic  Approach  to  Cybersecurity  (to  reflect  changes  in 
the  environment  since  its  creation  in  1998) 

Description:  With  this  option,  the  original  charter  of  the  NIAP  is  restored  but  updated  to 
address  changes  in  the  environment,  including  its  relationship  to  C&A  and  the  Federal 
cybersecurity  program.  Since  most  unintentional  vulnerabilities  are  caused  by  a  rela¬ 
tively  small  set  of  common  implementation  errors,  and  many  of  these  errors  can  be  de¬ 
tected  or  countered  by  tools,  employing  various  tools  should  improve  the  cost- 
effectiveness  of  the  evaluation  process  in  general.  This  update  provides  for  a  formaliza¬ 
tion  of  the  NIAP  as  an  entity  with  a  more  stable  funding  source,  separate  from  the  agency 
budgets  and  providing  a  detailed  set  of  requirements  together  with  oversight.  The  C&A 
processes  would  be  more  than  happy  to  take  advantage  of  product  evaluation  data,  which 
must  be  made  available  to  C&A  accreditors.  To  the  extent  that  NIAP/CCEVS  can  change 
their  documentation  and  evidence  to  also  meet  the  needs  of  system  security  certification 
agents  (and  system  auditors,  system  developers/integrators),  these  evaluation  deliverables 
may  prove  useful.  This  option  would  include  vulnerability  testing  in  all  evaluations  and 
the  development  in  test  methods  in  conjunction  with  PP  development  and  evaluation 
work  units. 

Additional  responsibilities  in  this  new  environment  would  include  a  more  aggressive 
partnership  with  industry  to  ensure  the  cost-effectiveness  of  this  effort,  and  an  aggressive, 
coherent  ET&A  effort.  This  option  could  institute  a  number  of  improvements  to  the 
evaluations  identified  as  current  shortfalls.  New  assurance  techniques  (peer  review, 
source  code  review  tools,  etc.)  would  be  evaluated  and  added  to  PPs  and  evaluation 
schemes.  The  modernized  NIAP  would  work  to  incorporate  these  techniques  into  the  CC. 
Personnel  in  laboratories  and  government  would  undergo  a  credentials  program. 

Pros:  This  option  directly  addresses  changes  in  the  cybersecurity  environment  that  have 
occurred  since  the  NIAP  was  established,  including  the  new  requirements  of  FISMA  and 
challenges  posed  by  foreign  software  development  and  foreign  chip  development,  by  re¬ 
quiring  the  use  of  tools  that  will  examine  these  issues  (such  as  nefarious  code  develop¬ 
ment).  The  ET&A  would  ensure  an  educated  consumer  that  was  identified  as  a  critical 
concern.  Maintenance  assurance  and  flaw  remediation  programs  would  maintain  product 
security  through  the  normal  software  development  cycles.  Increased  and  improved  test¬ 
ing  would  cover  some  aspects  of  the  concerns  for  screening  against  nefarious  or  mali¬ 
cious  code.  It  also  provides  a  more  cost-effective  process,  encouraging  industry  partici¬ 
pation. 

Cons:  Budget  requirements  would  have  to  be  resolved  through  the  agencies.  Although 
this  option  does  acknowledge  and  address  changes  in  the  cybersecurity  environment 
(Federal,  state/local  and  private  sector),  it  requires  policy,  responsibility  and  program¬ 
matic  changes  in  an  already  complicated  situation.  It  may  also  require  some  changes  to 
ET&A  programs  outside  of  the  NIAP  (DITSCAP,  etc.).  Additional  research  and  devel¬ 
opment  would  be  required  to  implement  this  option. 

7.2.5  Option  5:  Integrated  Approach  to  Cybersecurity 

Description:  This  Option  looks  at  a  state  beyond  the  current  thinking  of  what  the  NIAP 
was  originally  intended  to  be,  and  addresses  the  larger  issue  of  what  an  integrated  ap- 
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proach  should  be  to  ensure  secure,  functional  information  processes  for  the  Federal  gov¬ 
ernment  as  IT  continues  to  evolve  into  new  areas.  It  is  a  logical  extension  of  the  NIAP’s 
charter  to  look  at  not  only  product  evaluation  from  the  end-point,  but  at  software  assur¬ 
ance  in  developing  more  secure  and  reliable  products  and  then  ensuring  the  proper  con¬ 
figuration  and  operation  of  that  product  in  a  systems  environment.  Some  of  the  issues  to 
be  addressed  in  this  option  include:  advice  and  consent  for  products  in  the  C&A  process, 
and  development  of  more  comprehensive  and  robust  configuration  guidelines.  There  is  no 
guarantee  that  user  organizations  can/will  adhere  to  the  evaluation  environ¬ 
ments/configurations.  However,  even  if  evaluated  products  are  not  used  in  their  evalu¬ 
ated  environment/configuration,  C&A  must  be  performed.  It  is  unreasonable  to  expect 
the  evaluated  configuration  to  meet  the  needs  of  all  users  of  a  product.  C&A  is  per¬ 
formed  independent  of  whether  an  evaluated  product  is  used  in  a  system  and  is  independ¬ 
ent  of  whether  the  product  is  used  in  its  evaluated  configuration.  Using  evaluated  prod¬ 
ucts  in  a  system  should  not  be  construed  as  forcing  the  system  developers  to  implement 
the  system  in  the  evaluated  configurations.  In  fact,  when  using  several  evaluated  prod¬ 
ucts  in  a  system,  it  is  highly  unlikely  that  the  configuration  requirements  of  each  product 
can  be  met  simultaneously.  The  advice  and  consent  will  assist  in  evaluating  the  impact  of 
these  configuration  and  environment  changes.  This  option  would  also  address  the  NIAP 
participation  in  increased  and  focused  research  into  areas  directly  contributing  to  this  ef¬ 
fort. 

Pros:  This  option  acknowledges  that  the  environment  has  changed  since  the  original 
NIAP  charter  and  addresses  changes  required  by  that  new  environment.  It  would  recog¬ 
nize  assurance  techniques  that  are  not  yet  incorporated  into  the  CC,  while  at  the  same 
time  maintaining  international  mutual  recognition  where  possible.  The  ET&A  would  en¬ 
sure  an  educated  consumer  that  was  identified  as  a  critical  concern.  Maintenance  assur¬ 
ance  and  flaw  remediation  programs  would  maintain  product  security  through  the  normal 
software  development  cycles.  Increased  and  improved  testing  would  cover  some  aspects 
of  the  concerns  for  screening  against  nefarious  or  malicious  code.  It  also  provides  a  more 
cost-effective  process,  encouraging  industry  participation. 

Cons:  Budget  requirements  would  have  to  be  resolved  through  the  agencies.  This  option 
may  require  significant  changes  in  policy  and  practice  outside  of  the  NIAP  itself  that  may 
present  overwhelming  challenges  to  existing  organizations  unless  their  responsibilities 
are  re-scoped.  Using  additional  assurance  techniques  requires  evaluating  how  these  tech¬ 
niques  interact  with  the  techniques  already  in  use.  Training  would  have  to  be  provided  to 
the  evaluation  laboratories.  Changes  to  the  CC  would  require  international  consensus 
and  may  take  time.  Automated  source  code  review  may  be  a  tricky  issue  with  respect  to 
intellectual  property  rights.  Vulnerability  testing  and  source  code  review  would  be 
greatly  facilitated  by  tool  development.  It  may  also  require  some  changes  to  ET&A  pro¬ 
grams  outside  of  the  NIAP  (DITSCAP,  etc.).  Additional  research  and  development  would 
be  required  to  implement  this  option. 

7.2.6  Option  6:  Forward  Looking  Approach  to  Cybersecurity  (new  paradigm) 

Description:  The  sixth  option  is  to  move  the  approach  to  security  evaluation  to  a  new 
paradigm  for  operation  and  evaluation  that  would  better  address  the  wide-ranging 
changes  that  have  occurred  in  the  environment  and  community  that  the  NIAP  must  serve. 
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This  is  not  a  change  in  the  NIAP  following  the  progression  already  described,  but  a 
whole  new  way  of  thinking  about  the  problem  of  cybersecurity.  While  it  is  not  possible 
to  describe  this  option  and  associated  resource  requirements  in  actionable  detail  without 
further  study,  several  aspects  of  this  option  are  clear  at  this  time. 

The  new  paradigm  must  address,  in  a  holistic  manner,  risks  and  vulnerabilities  to  sys¬ 
tems.  It  must  ensure  that  the  security  of  a  system  is  greater  than  the  sum  of  its  parts.  It 
must  also  address  issues  related  to  changes  in  system  ownership,  rapid  changes  in  system 
composition,  changes  in  data  ownership  and  location,  changes  in  user  expertise,  and  in¬ 
creasingly  complex  systems.  In  other  words,  the  new  paradigm  must  be  as  dynamic  as  the 
environment  within  which  it  must  work. 

Clearly,  there  are  many  approaches  for  specifying  and  assessing  security  functionality 
other  than  using  the  CC.  For  example,  the  open  source  approach  of  having  many  inde¬ 
pendent  observers  perfonn  code  reviews  would  perhaps  ease  fears  of  trap  doors  and  Tro¬ 
jan  horses.  Alternatively,  evaluators  could  observe  a  product  producer’s  methodology,  in 
lieu  of  assessing  documentation.  Other  schemes  may  include  active  lab  vulnerability  test¬ 
ing  for  all  new  identified  vulnerability  threats  (viruses,  Trojan  horses,  etc.).  NIAP  is  not 
currently  disposed  to  require  these  types  of  tests,  although  they  have  considered  it  and 
even  drafted  (but  not  approved)  some  policy  in  this  area. 

Pros:  This  option  acknowledges  that,  even  with  changes  that  address  the  current  envi¬ 
ronment,  innovative  thinking  will  be  required  to  keep  ahead  of  the  threats  and  vulner¬ 
abilities  in  cybersecurity.  Adoption  of  this  option  would  ensure  that  cybersecurity 
evaluations  remain  relevant  in  a  dynamic  environment. 

Cons:  This  option  does  not  solve  the  near  tenn  problems  with  the  NIAP,  so  it  must  be 
pursued  in  combination  with  other  options  described  above.  There  is  also  technical  risk 
with  this  option  since  nothing  in  the  current  DARPA  or  NSF  research  pipelines  that  is  ad¬ 
dressing  a  new  paradigm26. 

7 . 3  Examining  the  Options  in  the  Performance/Cost  Trade  Space 

The  key  to  providing  perfonnance/cost  trade  space  options  is  in  having  good  data.  We 
have  seen,  however,  that  a  metrics  program  was  never  put  in  place.  There  are  no  per¬ 
formance  statistics  to  say  how  well  the  cybersecurity  approaches  are  doing  in  general, 
and  with  and  without  product  evaluation  under  the  current  process.  Nor,  is  there  accurate 
cost  data  because  of  the  commercial  laboratory  system  and  the  treating  of  costs  and 
budgets  as  proprietary  data.  The  first  priority  in  any  of  the  options  that  seek  to  improve 


In  testimony  to  the  US  House  of  Representatives  Committee  on  Science  in  May  2003,  DARPA  Director  Dr.  Tony  Tether  described 
DARPA's  past  investments  in  information  assurance  and  cybersecurity  research.  This  work  included  development  and  improvement 
of  firewalls,  intrusion  detection  methods,  and  intrusion  tolerance  techniques  that  allow  systems  to  operate  through  attacks.  The  bulk  of 
this  research,  from  DARPA's  point  of  view,  has  been  completed  and  DARPA  has  moved  on  to  more  advanced  concepts  of  cognitive 
computing  in  which  computer  systems  are  expected  to  know  what  they  are  doing  —  including  knowing  when  they  are  under  attack  and 
how  to  respond.  DARPA's  original  firewall  research  has  matured  into  widely  used  commercial  products.  The  remainder  of  this  re¬ 
search  is  still  in  the  proof  of  concept  and  product  development  pipeline. 

Also  in  2003,  NSF  started  a  process  to  reinvigorate  their  Cyber  Trust  program.  In  2004,  NSF  funded  50  research  projects  addressing 
different  aspects  of  computer  and  network  security,  privacy,  and  trust.  Virtually  all  of  this  research  focuses  on  fundamental  research 
questions  that  have  long-range  implications.  This  work,  however,  is  not  intended  to  address  today's  immediate  computer  security 
problems.  Research  results  that  can  be  transitioned  quickly  into  products  and  applications  are  good,  but  this  is  not  a  criteria  NSF  uses 
in  selecting  research  projects  for  funding. 
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performance  is  to  do  the  research  necessary  to  establish  metrics  (including  costs)  and  to 
place  a  metrics  program  in  place.  Further,  as  delineated  in  Option  6  an  annual  report  and 
an  independent  assessment  should  be  provided  until  there  is  enough  data  to  properly  un¬ 
derstand  how  the  actions  needed  to  achieve  the  various  options  provide  the  trade-off  be¬ 
tween  cost  and  performance.  These  analyses  should  be  part  of  the  near  term  efforts  and 
be  completed  before  proceeding  to  mid-term  efforts.  Once  metrics  exist,  thresholds  and 
requirements  can  be  derived.  This  directly  applies  to  options  3  through  6.  None-the-less, 
a  notional  representation  of  that  trade  space  is  presented  in  Figure  7.  This  notional  repre¬ 
sentation  was  developed  by  the  analysts  to  illustrate  how  these  changes  may  affect  the 
performance/cost  trade  space.  In  this  figure,  rough  estimates  based  upon  experience  and 
a  little  analysis,  is  presented  to  provide  a  “feel”  for  the  options  outlined  in  this  chapter 
and  presented  in  detail  in  the  next  chapter.  These  data  are  unreliable  and  the  metrics  pro¬ 
gram  should  be  the  first  indication  of  their  correctness  in  terms  of  trend. 
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Option  1  Eliminate  NIAP 
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Option  2  No  Change 

Option  3  Restore  NIAP 

Option  4  Modernized  Approach 

— 

Option  5  Integrated  Approach 

Option  6  Forward  looking  Approach 

(In  conjunction  with  Ootions  3-5) 

Forward  Looking  Operating. Curve-  ' ' 

Layered  Approach  OperatingXurve" '  ‘ 


Added  Product  Evaluation.  Operating' Curve 

Option  1 


Stand  Alone  System  Certification- 


Option  2  ' 


Cost 


Figure  7.  Notional  Performance/Cost  Trade  Space  for  the  6  Options  Presented 
7 . 4  Summary 

This  chapter  has  attempted  to  provide  a  series  of  options  that  address  the  issues  raised  in 
previous  chapters  and  how  those  issues  might  be  addressed,  from  maintaining  the  status 
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quo  to  the  most  radical  new  thinking.  What  is  apparent  from  the  analysis  presented  in  the 
prior  chapters  is  a  feeling  in  the  community  at  large  that  the  status  quo  (Option  2)  is  not 
meeting  expectations  and  that  something  must  change.  Whether  the  change  is  to  discon¬ 
tinue  requiring  product  evaluations  for  the  lower  assurance  levels  or  improve  the  process, 
consideration  must  be  given  to  the  changes  required  in  policy,  processes  and  resources  so 
that  an  informed  decision,  with  an  understanding  of  the  costs  and  benefits  and  ultimate 
consequences,  may  be  made.  The  areas  where  expectations  are  not  met  can  generally  be 
improved  with  an  increased  emphasis  (e.g.,  funding)  or  a  new  requirement  for  evaluation. 
Many  expectations  are  conflicting  and  cannot  all  be  met.  It  is  also  noted  that  the  gov¬ 
ernment  is  not  responsible  for  meeting  all  of  the  cyberspace  requirements,  but  should 
place  a  structure  in  place  that  raises  the  level  for  all  concerned  without  undue  burden  on 
any  sector.  Chapter  8  builds  on  these  options  by  providing  a  roadmap  for  improving  the 
NIAP  program. 
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8 .  Roadmaps  for  Accomplishing  Options 


The  preceding  chapters  have  provided  recommendations  and  options  without  context. 
They  have  indicated  individual  discrepancies  in  the  current  system  with  an  indication  of 
what  is  needed  to  repair  them.  Reference  tags  are  used  when  findings  and  recommenda¬ 
tions  would  otherwise  be  repeated.  After  each  action  to  be  undertaken  for  an  option  are 
tags  to  related  findings  or  recommendations.  Annex  I  is  an  alphabetical  listing  of  the  tags, 
showing  the  chapter,  section,  and  approximate  page  number  where  the  finding  or  recom¬ 
mendation  appears.  This  enables  the  reader  to  trace  back  to  the  supporting  evidence  for 
each  option. 

The  purpose  of  this  chapter  is  to  provide  a  coherent  approach  for  implementing  each  of 
the  options.  It  should  be  noted  that  many  of  the  actions  in  the  options  are  beyond  the 
functionality  that  NIAP  was  originally  intended  to  undertake.  Indeed  many  may  be  out¬ 
side  of  NIAP  control  and  require  input  or  actions  from  other  agencies,  or  legislative  ac¬ 
tion. 

8 , 1  Option  Roll-out 

Options  2-5  form  a  set  of  increasing  requirements.  As  such,  exercise  of  the  option  ele¬ 
ments  can  fit  budgetary  constraints  initially;  however,  longer-tenn  commitments  are 
made  as  the  option  number  increases.  The  options  and  the  relation  to  each  other  are 
shown  in  the  Table  below. 


Table  19.  Relationship  of  the  Various  Options 


Option 

Title 

Details  ap¬ 
pear  in 

Actions  Required 

Near-term 

Mid-term 

Long-Term 

1 

Eliminate  the 
NIAP 

8.2 

Elimination 

2 

Continue  the 
Current  NIAP 

8.3 

1.  Trained  Personnel. 

2.  Requirements. 

3.  Security  Support 
Group. 

4.  ISO/CC.  Cost 
Reduction.  Stan¬ 
dards. 

3 

Restore  the 

NIAP 

8.4 

1 .  Option  2+ 

2.  Requirements. + 

1 .  C&A  Interface. 

2.  NIAP  Process  Im¬ 
provement. 
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Option 

Title 

Details  ap¬ 
pear  in 

Actions  Required 

Near-term 

Mid-term 

Long-Term 

4 

Modernized 
Approach  to 
Cybersecurity 

8.5 

1.  Option  3+ 

2.  Formalization. 

3.  Testing. 

4.  Security  Support 
Group.  + 

5.  NIAP  Process 
Improvement. 

1.  Option  3+ 

2.  C&A  Interface.  + 

3.  NIAP  Process  Im¬ 
provement.  + 

4.  ISO/CC.  Cost  Reduc¬ 
tion.  Standards. 

5.  Consolidation. 

6.  Security  Support 

1 .  Cost  Reduc¬ 
tion. 

2.  Consolidation. 

3.  ISO/CC. 

Cost  Reduc¬ 
tion.  Stan¬ 
dards. 

5 

Integrated  Ap¬ 
proach  to  Cy¬ 
bersecurity 

8.6 

Option  4 

1 .  Option  4  + 

2.  C&A  Interface.  + 

Option  4 

6 

Forward  looking 
Approach  to 
Cybersecurity 

8.7 

Independent  Assess¬ 
ment 

Independent  As¬ 
sessment 

In  an  ideal  world,  budgets  would  allow  us  to  do  through  option  4  which  represents  and 
modernization  of  the  NIAP  process  to  fit  the  world  changes  that  have  evolved  since  its 
inception.  Option  5  adds  to  this,  but  no  significant  additions  exist  until  the  mid-term,  al¬ 
lowing  a  rollout  and  evaluation  of  Option4/5  until  the  mid-term,  at  which  point  a  choice 
can  be  made.  Option  6  should  be  implemented  in  any  event  because  it  will  monitor  and 
provide  updates  and  coordination  of  cybersecurity  issues. 


8 . 2  Option  1:  Eliminate  the  NIAP 


Option 

Title 

Details  ap¬ 
pear  in 

Actions  Required 

Near-term 

Mid-term 

Long-Term 

1 

Eliminate  the 
NIAP 

8.2 

Elimination 

Eliminating  the  NIAP  is  not  recommended  [FN-1,  RN-1],  It  is  recommended  that  the 
NIAP,  as  an  entity  be  retained.  The  NIAP  has  many  shortcomings  that  need  to  be  ad¬ 
dressed,  but  its  strength  is  in  the  gathering  of  expertise,  the  development  of  an  infrastruc¬ 
ture,  and  its  programmatic  ties  to  the  international  community.  While  a  new  organization 
or  approach  could  be  developed,  the  administrative  burden  of  recreating  these  strengths  is 
large  and  without  guarantee  of  greater  success.  Having  said  that,  the  NIAP  should  be  re¬ 
tained,  it  is  recommended  that  it  immediately  be  strengthened  and  improved  and  moved 
towards  a  process  that  is  more  responsive  to  stakeholders  and  the  nation’s  needs,  and  fits 
better  within  the  architecture  of  software  assurance  processes. 

Options  2  to  5  assume  that  a  properly  developed  and  integrated  product  evaluation  is  nec¬ 
essary  to  overall  program  of  cybersecurity.  This  overall  program  includes  specific  algo¬ 
rithm  evaluation  overseen  by  NIST  (as  in  FIP-140)  and  system  level  evaluations  as  pre¬ 
scribed  in  DITSCAP,  NIACAP  and  OMB/FISMA. 


8 . 3  Option  2:  Continue  the  NIAP  (in  its  current  form) 


Option 

Title 

Details  ap¬ 
pear  in 

Actions  Required 

Near-term 

Mid-term 

Long-Term 

2 

Continue  the 
Current 

NIAP 

8.3 

1.  Trained  Personnel. 

2.  Requirements. 

3.  Security  Support 
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Group. 

4.  ISO/CC.  Cost 
Reduction.  Stan¬ 
dards. 

If  Option  2  is  undertaken,  then  the  following  actions  need  to  be  taken  as  quickly  as  possi¬ 
ble: 


2-1  near-tenn. 


2-2  near-tenn. 


2-3  near-tenn. 


2-4  near-tenn. 


Trained  Personnel.  Establish  a  program  to  increase  the  number  of 
educated,  trained  evaluation  personnel.  [RPEta-1,  FN-2,  AN -22,  FEPe- 
1,  ES-06] 

Requirements,  to  include  some  NIST  participation,  and  adequate  re¬ 
sources  for  the  evaluation  function.  [RPCy-2,  FPCy-2,  RPCy-3,  ENi-1, 
ENi-2] 

Security  Support  Group.  Set  up  a  security  support  group  (SSG)  to 
provide  legal/guidance  and  policy  consolidation  services  (legal  clear¬ 
inghouse  activities) .  [FPCy- 1 ,  RPCy- 1  ] 

ISO/CC.  Cost  Reduction.  Standards.  Push  ISO/CC  to  accept  ven¬ 
dor  documentation  in  lieu  of  CC  developed  evidence  (cost  reduction). 
[AN-11,  OCT- 1,  OCT-2] 


8 . 4  Option  3:  The  NIAP  Restored  to  the  Original  Intent 


Option 

Title 

Details  ap¬ 
pear  in 

Actions  Required 

Near-term 

Mid-term 

Long-Term 

3 

Restore  the 
NIAP 

8.4 

1 .  Option  2+ 

2.  Requirements. + 

1 .  C&A  Interface. 

2.  NIAP  Process  Im¬ 
provement. 

If  Option  3  is  undertaken,  then  the  following  actions  should  be  taken  as  quickly  as  possi¬ 
ble: 


3-1  neartenn.  Trained  Personnel.  Establish  a  program  to  increase  the  number  of 
educated,  trained  evaluation  personnel.  [AN-22,  RPEta-1,  FN-2,  FEPe- 
1,  ES-06] 

27 

3-2  near  tenn.  Requirements. 

1.  to  include  a  fully  participating  NIST  and  potentially  new 
partners  such  as  DHS.  Other  potentially  new  partners  such  as 
Consortia  and  academia  would  provide  their  own  funding. 

[RPSt-1] 

2.  Funding  should  be  applied  to  research,  metrics  (including  ROI 
metrics),  tool  development  (source  code  scanners,  port  sniffers, 
test  benches  for  specific  vulnerabilities,  etc.  -  not  on  tools  to 


~7  Text  in  italics  indicates  new  actions  not  found  in  lower  numbered  options. 

28  In  all  aspects  of  NIAP,  including  research,  tools,  education  and  CCEVS  as  well  as  NVLAP. 
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help  document  evaluation),  [RPCy-3,  FPRe-1,  FPRe-5,  RPRe- 
2,  FN-2,  FERe-1,  ES-12] 

3.  Metrics  should  be  tracked  an  analyzed,  at  least  annually  until  a 
clear  picture  of  the  worth  of  other  action  may  be  perceived. 

4.  development  and  dissemination  of  training  materials  for  devel¬ 
opers,  managers,  users,  consumers  and  others  not  involved  in 
the  evaluation  process,  [FN-6,  FPEta-11,  FPEta-2,  RPEta-1, 
FEKn-2,  AN- 17,  ES-01] 

5.  CC  Standards  participation  and  PP  development  as  well  as 
evaluations.  [FEPP-1,  RN-1,  ES-04,  ES-05] 

29 

(finish  mid-term,  start  near-term). 

3-3  near  tenn.  Security  Support  Group.  Set  up  a  security  support  group  (SSG)  to 
provide  legal/guidance  and  policy  consolidation  services  (legal  clear¬ 
inghouse  activities) .  [FPCy- 1 ,  RPCy- 1  ] 

3-4  near  tenn.  ISO/CC.  Cost  Reduction.  Standards.  Push  ISO/CC  to  accept  ven¬ 
dor  documentation  in  lieu  of  CC  developed  evidence  (cost  reduction). 
[AN-11,  OCT- 1,  OCT-4] 

If  Option  3  is  undertaken,  then  these  items  should  be  begun  as  soon  as  they  can  be  practi¬ 
cably  started: 

3-1  mid-term.  C&A  Interface.  Make  evaluation  technical  reports  and  testing  data 
releasable  to  C&A  authorities.  [RPAq-3,  FECa-1] 

3-2  mid-term.  Improvement  of  the  NIAP  Processes  .  Develop  publicly  available 
tools,  a  NIST  web  site  for  distribution  and  maintenance,  and  tasking  to 
keep  up  to  date.  [AN-08,  FETe-1] 


8 . 5  Option  4:  Modernized  Approach  to  Cybersecurity 


Option 

Title 

Details  ap¬ 
pear  in 

Actions  Required 

Near-term 

Mid-term 

Long-Term 

4 

Modernized 
Approach  to 
Cybersecurity 

8.5 

1.  Option  3+ 

2.  Formalization. 

3.  Testing. 

4.  Security  Support 
Group.  + 

5.  ISOKC+ 

6.  NIAP  Process  Im¬ 
provement. 

1.  Option  3+ 

2.  C&A  Interface.  + 

3.  NIAP  Process  Im¬ 
provement.  + 

4.  ISO/CC. 

Cost  Reduction. 
Standards. 

5.  Consolidation. 

6.  Security  Support 
Group. 

1 .  Cost  Reduction. 

2.  Consolidation. 

3.  ISO/CC. 

Cost  Reduction. 
Standards. 

If  Option  4  is  undertaken,  then  the  following  items  should  be  done  as  quickly  as  possible: 


29  Additional  requirements  from  previous  options  are  shown  in  italics. 
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4-1  near-term. 


4-2  near-tenn. 


4-3  near-tenn. 


4-4  near-term. 


4-5  near-tenn. 


4-6  near-tenn. 


Trained  Personnel.  Establish  a  program  to  increase  the  number  of 
educated,  trained  evaluation  personnel.  [AN-22,  RPEta-1,  FN-2,  FEPe- 
1,  ES-06] 

Formalization.  Immediately  formalize  the  NIAP  by  providing  a  fund¬ 
ing  line  and  specific  requirements  associated  with  its  intended  pur¬ 
poses.  [RPCy-2] 

Requirements. 

1.  to  include  a  fully  participating  NIST  and  potentially  new 
partners  such  as  DHS.  Other  potentially  new  partners  such 
as  Consortia  and  academia  would  provide  their  own  fund¬ 
ing.  [RPSt-1] 

2.  Funding  should  be  applied  to  research,  metrics  (including 
ROI  metrics),  tool  development  (source  code  scanners,  port 
sniffers,  test  benches  for  specific  vulnerabilities,  etc.  -  not 
on  tools  to  help  document  evaluation),  [RPCy-3,  FPRe-1, 
FPRe-5,  RPRe-2,  FN-2,  FERe-1,  ES-12] 

3.  Metrics  should  be  tracked  an  analyzed,  at  least  annually  un¬ 
til  a  clear  picture  of  the  worth  of  other  action  may  be  per¬ 
ceived.  [FN-6,  FPEta-11,  FPEta-2,  RPEta-1,  FEKn-2,  AN- 
17,  ES-01] 

4.  development  and  dissemination  of  training  materials  for 
developers,  managers,  users,  consumers  and  others  not  in¬ 
volved  in  the  evaluation  process,  [FN-6,  FPEta-11,  FPEta- 
2,  RPEta-1,  FEKn-2,  AN- 17,  ES-01] 

5.  CC  Standards  participation  and  PP  development  as  well  as 
evaluations.  [FEPP-1,  RN-1,  ES-04,  ES-05] 

(finish  mid-term,  start  near-term). 

Testing.  Require  all  evaluations  to  undergo  vulnerability  analysis, 
testing  as  well  as  assurance  maintenance  and  flaw  remediation  (the 
latter  two  are  not  now  a  requirement  in  any  assurance  packages30 ,  and 
little  specific  application  of  testing  and  vulnerability  analysis  are  re¬ 
quired  at  EAL4  and  below).  [AN-07,  FN-6,  FETe-3,  FEAM-1] 

Security  Support  Group.  Set  up  a  security  support  group  (SSG)  to 
provide  legal/guidance  and  policy  consolidation  services  (legal  clear¬ 
inghouse  activities).  The  SSG  will  also  provide  analysis  of  cybersecu- 
ritv  research  in  government,  industry,  and  academia.  [FPCy-1,  RPCy- 
4,  FPRe-2,  FPRe-3] 

ISO/CC.  Standards.  Push  ISO/CC  to  include  test  in  CEM  and  PPs. 
Push  ISO/CC  to  reduce  evaluation  paperwork  requirements.  [FETe-2, 
AN-11,  OCT- 1,  OCT-4] 


30  DoD  may  be  addressing  each  of  these  as  cited  previously  in  the  text,  but  a  more  formal  and  wider  appli¬ 
cation  is  needed  for  DHS  considerations,  (see  section  8.8) 


103 


4-7  near-term.  Improvement  of  the  NIAP  Processes.  Develop  a  personnel  certifica¬ 
tion  process  and  credential  all  evaluators  and  validators  (finish  mid¬ 
term,  start  near-term).  [AN -22,  FEPe-1] 


If  Option  4  is  undertaken,  then  these  items  should  be  begun  as  soon  as  they  can  be  practi¬ 
cably  started: 


4-1  mid-term. 
4-2  mid-term. 
4-3  mid-term. 

4-4  mid-term. 

4-5  mid-term. 

4-6  mid-term. 

4-7  mid-term. 

4-8  mid-term. 


C&A  Interface.  Make  evaluation  technical  reports  and  testing  data 
releasable  to  C&A  authorities.  [RPAq-3,  FECa-1] 

C&A  Interface.  Modify  C&A  under  OMB,  DITSCAP,  NIACAP,  etc.  to 
allow  reuse  of  the  product  evaluation  data.[  FPAq-2,  RPAq-3] 
Improvement  of  the  NIAP  Processes.  Develop  publicly  available 
tools,  a  NIST  web  site  for  distribution  and  maintenance,  and  tasking  to 
keep  up  to  date.  Require  that  tools  be  run  against  all  evaluated  prod¬ 
ucts.  [AN-08,  FETe-1] 

Improvement  of  the  NIAP  Processes.  Develop  a  personnel  certifica¬ 
tion  process  and  credential  all  evaluators  and  validators  (finish  mid¬ 
term,  start  near-term) .  [AN-22,  FEPe-1,  ES-06] 

ISO/CC.  Cost  Reduction.  Standards.  Push  ISO/CC  to  require  tools 
run  against  all  evaluated  products.  [AN-07,  AN-09,  RN-1,  FETe-2, 
FETe-3] 

Consolidation.  Bring  all  of  Federal  government  under  the  product 
evaluation  mandate  similar  to  NSTISSIP  (but  not  until  tools  are  avail¬ 
able  and  a  stable  set  of  PPs  are  developed  at  all  levels).  -  Legislative 
and  policy  issues.  [FPSt-2,  FPAq-1,  RPAq-1,  RPAq-2] 

Security  Support  Group.  Institute  ROI  studies  by  the  SSG,  using  the 
metrics  developed  under  near-term  4-3.  [FPRe-4,  FPRe-5,  RPRe-2, 
AN- 16,  AN-23] 

Adjust  based  on  mid-term  4-7. 


If  Option  4  is  undertaken,  then  these  items  are  follow-on  and  should  be  started  at  appro¬ 
priate  times  indicated  by  completion  of  other  items  on  the  lists  above: 


4-1  long-term. 


4-2  long-term. 


4-3  long-term. 


Cost  Reduction.  After  item  mid-term  4-3  is  complete,  set  up  an  alter¬ 
nate  assurance  process  for  products  of  basic  security  that  includes 
only  the  tools,  security  function  verification,  and  vendor  process 
evaluation  (goal  -  cost  effective  evaluations  that  raise  the  bar  for  eve¬ 
ryone).  Equivalent  EAL3-4  and  above  to  be  done  through  Common 
Criteria.  [EAa-1,  OAa-1] 

Consolidation.  Include  critical  infrastructure  under  this  mandate.  - 
Legislative  and  policy  issues  need  to  be  resolved.  [ECI-1,  ECI-2,  ECI- 
3,  OCI-1]  Include  selected  product  classes  outside  of  the  normal  IA  or 
IA-enabled product  type.  [AN-27] 

ISO/CC.  Cost  Reduction.  Standards.  Push  CC  to  this  level  (de¬ 
scribed  in  4-1  long-term).  [AN-20] 
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8 . 6  Option  5:  Integrated  Approach  to  Cybersecurity 


Option 

Title 

Details  ap¬ 
pear  in 

Actions  Required 

Near-term 

Mid-term 

Long-Term 

5 

Integrated 
Approach  to 
Cybersecurity 

8.6 

Option  4 

1.  Option  4  + 

2.  C&A  Interface.  + 

Option  4 

If  Option  5  is  undertaken,  then  the  following  items  should  be  done  as  quickly  as  possible: 


5-1  near-tenn. 


5-2  near-tenn. 


5-3  near-tenn. 


5-4  near-tenn. 


5-5  near-tenn. 


Trained  Personnel.  Establish  a  program  to  increase  the  number  of 
educated,  trained  evaluation  personnel.  [AN-22,  RPEta-1,  FN-2,  FEPe- 
1,  ES-06] 

Formalization.  Immediately  fonnalize  the  NIAP  by  providing  a  fund¬ 
ing  line  and  specific  requirements  associated  with  its  intended  pur¬ 
poses.  [RPCy-2] 

Requirements. 

1.  to  include  a  fully  participating  NIST  and  potentially  new 
partners  such  as  DHS.  Other  potentially  new  partners  such 
as  Consortia  and  academia  would  provide  their  own  fund¬ 
ing.  [RPSt-1] 

2.  Funding  should  be  applied  to  research,  metrics  (including 
ROI  metrics),  tool  development  (source  code  scanners,  port 
sniffers,  test  benches  for  specific  vulnerabilities,  etc.  -  not 
on  tools  to  help  document  evaluation),  [RPCy-3,  FPRe-1, 
FPRe-5,  RPRe-2,  FN-2,  FERe-1,  ES-12] 

3.  Metrics  should  be  tracked  an  analyzed,  at  least  annually  un¬ 
til  a  clear  picture  of  the  worth  of  other  action  may  be  per¬ 
ceived.  [FN-6,  FPEta-11,  FPEta-2,  RPEta-1,  FEKn-2,  AN- 
17,  ES-01] 

4.  NIAP-funded  evaluations  for  small  business  and  open 
source  software  where  the  developers  cannot  fund  the 
evaluation  (possibly  requiring  free/low-cost  use/support, 
modification  rights,  and/or  specialized  changes),  [OCT-8, 
ECT-1,  ECT-2] 

5.  development  and  dissemination  of  training  materials  for 
developers,  managers,  users,  consumers  and  others  not  in¬ 
volved  in  the  evaluation  process,  [FN-6,  FPEta-11,  FPEta- 
2,  RPEta-1,  FEKn-2,  AN- 17,  ES-01] 

6.  CC  Standards  participation  and  PP  development  as  well  as 
evaluations.  [FEPP-1,  RN-1,  ES-04,  ES-05] 

(finish  mid-term,  start  near-term). 

Testing.  Require  all  evaluations  to  undergo  vulnerability  analysis, 
testing  as  well  as  assurance  maintenance  and  flaw  remediation  (the  lat¬ 
ter  two  are  not  now  a  requirement  in  any  assurance  packages,  and  little 
specific  application  of  testing  and  vulnerability  analysis  are  required  at 
EAL4  and  below).  [AN-07,  FN-6,  FETe-3,  FEAM-1] 

Security  Support  Group.  Set  up  a  security  support  group  (SSG)  to 
provide  legal/guidance  and  policy  consolidation  services  (legal  clear- 
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inghouse  activities).  The  SSG  will  also  provide  analysis  of  cybersecu¬ 
rity  research  in  government,  industry,  and  academia.  [FPCy-1,  RPCy- 
4,  FPRe-2,  FPRe-3] 

5-6  near-tenn.  ISO/CC.  Cost  Reduction.  Standards.  Push  ISO/CC  to  include  test 
in  CEM  and  PPs.  Push  ISO/CC  to  reduce  evaluation  paperwork  re¬ 
quirements.  [FETe-2,  AN-11,  OCT-1,  OCT-4] 

5-7  near-tenn.  Improvement  of  the  NIAP  Processes.  Develop  a  personnel  certifica¬ 
tion  process  and  credential  all  evaluators  and  validators  (finish  mid- 
tenn,  start  near-term).  [AN-22,  FEPe-1] 

If  Option  5  is  undertaken,  then  these  items  should  be  begun  as  soon  as  they  can  be  practi¬ 
cably  started: 

5-1  mid-term.  C&A  Interface.  Make  evaluation  technical  reports  and  testing  data 
releasable  to  C&A  authorities.  [RPAq-3,  FECa-1 
5-2  mid-term.  C&A  Interface.  Institute  a  program  of  advice  and  consent,  where  the 
NIAP  participates  in  C&A  and  advises  on  the  use,  configuration,  and 
environment  for  evaluated  products,  as  well  as  other  functions  as  de¬ 
lineated  in  Chapter  6.  [FPAq-2,  RPAq-3] 

5-3  mid-term.  C&A  Interface.  Modify  C&A  under  OMB,  DITSCAP,  NIACAP,  etc. 

to  allow  reuse  of  the  product  evaluation  data,  and  recommend  the 
NIAP  Advise  and  Consent  of  element  2.  [FPAq-2,  RPAq-3] 

5-4  mid-term.  Improvement  of  the  NIAP  Processes.  Develop  publicly  available 
tools,  a  NIST  web  site  for  distribution  and  maintenance,  and  tasking  to 
keep  up  to  date.  Require  that  tools  be  run  against  all  evaluated  prod¬ 
ucts.  [AN-08,  FETe-1] 

5-5  mid-term.  Improvement  of  the  NIAP  Processes.  Develop  a  personnel  certifica¬ 
tion  process  and  credential  all  evaluators  and  validators  (finish  mid¬ 
term,  start  near-tenn).  [AN-22,  FEPe-1,  ES-06] 

5-6  mid-term.  ISO/CC.  Standards.  Push  ISO/CC  to  require  tools  run  against  all 
evaluated  products.  [AN-07,  AN-09,  RN-1,  FETe-2,  FETe-3] 

5-7  mid-term.  Consolidation.  Bring  all  of  Federal  government  under  the  product 
evaluation  mandate  similar  to  NSTISSIP  (but  not  until  tools  are  avail¬ 
able  and  a  stable  set  of  PPs  are  developed  at  all  levels).  -  Legislative 
and  policy  issues.  [FPSt-2,  FPAq-1,  RPAq-1,  RPAq-2 
5-8  mid-term.  Security  Support  Group.  Institute  ROI  studies  using  the  metrics  de¬ 
veloped  under  near-term  5-3.  [FPRe-4,  FPRe-5,  RPRe-2,  AN-16,  AN- 
23] 

5-9  mid-term.  Adjust  based  on  mid-tenn  5-8. 

If  Option  5  is  undertaken,  then  these  items  are  follow-on  and  should  be  started  at  appro¬ 
priate  times  indicated  by  completion  of  other  items  on  the  lists  above: 

5-1  long-tenn.  Cost  Reduction.  After  mid-term  5-3  is  complete,  set  up  an  alternate 
assurance  process  for  products  of  basic  security  that  includes  only  the 
tools,  security  function  verification,  and  vendor  process  evaluation 
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(goal  -  cost  effective  evaluations  that  raise  the  bar  for  everyone). 
Equivalent  EAL3-4  and  above  to  be  done  through  Common  Criteria. 
[EAa-1,  OAa-1] 

5-2  long-tenn.  Consolidation.  Include  critical  infrastructure  under  this  mandate.  - 
Legislative  and  policy  issues  need  to  be  resolved.  Include  selected 
product  classes  outside  of  the  normal  IA  or  IA-enabled  product  type. 
[AN-27] 

5-3  long-tenn.  ISO/CC.  Cost  Reduction.  Standards.  Push  CC  to  this  level  (de¬ 
scribed  in  long-tenn  5-1).  [AN-20] 


8.7  Option  6:  Forward  looking  Approach  to  Cyber  security 


Option 

Title 

Details  ap¬ 
pear  in 

Actions  Required 

Near-term 

Mid-term 

Long-Term 

6 

Forward 
looking  Ap¬ 
proach  to 
Cybersecurity 

8.7 

Independent  Assess¬ 
ment 

Independent  Assess¬ 
ment 

Option  6  does  not  stand  alone,  because  it  does  not  have  an  immediate  process  for  replac¬ 
ing  options  3-5.  However,  Option  6  does  not  apply  nor  should  it  be  used  in  conjunction 
with  Options  1  or  2.  If  Option  6  is  undertaken,  then  the  following  items  should  be  done 
as  quickly  as  possible: 


6-1  near-term. 
6-2  near-tenn. 


6-3  near-term. 
6-4  near-term. 


Instantiate  one  of  the  Options  3-5  in  the  interim. 

Independent  Assessment.  Provide  independent  and  periodic  audits  of 
the  cybersecurity  process.  This  should  include  tabulation  and  analysis 
of  the  metrics  developed  under  the  option  chosen. 

Independent  Assessment.  Provide  for  an  annual  report  for  the  state  of 
nation  s  cyberspace  security. 

Security  Support  Group.  Standards.  Augment  the  CC  interface  with 
related  standards  works. 


If  Option  6  is  undertaken,  then  these  items  are  follow-on  and  should  be  started  at  appro¬ 
priate  times  indicated  by  completion  of  other  items  on  the  lists  above: 

6-1  long-term.  Independent  Assessment.  Provide  for  the  overall  coordination  of  the 
cyberspace  activities  and  provide  an  arbitrage  sendee  where  the  needs 
of  various  elements  ( i.  e. ,  C&A  and  or  product  evaluation)  come  into 
conflict. 

Independent  Assessment.  Institute  new  approaches  as  reported  by  the 
above. 

8 . 8  Amplifying  Comments  for  specific  action  items. 

8.8.1  Trained  Personnel 

The  need  for  trained  personnel  is  an  underlying  problem  that  must  be  addressed  for  any 
of  the  viable  options.  As  described  in  Chapter  4,  the  NIAP  relies  upon  a  core  group  of 
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highly  educated,  trained,  and  experienced  experts  in  both  cybersecurity  evaluation  and 
standardization.  The  growth  of  the  IT  applications  in  general,  combined  with  increased 
connectivity  and  complexity  of  security  issues  will  overwhelm  this  core  group.  As  it  is 
the  NIAP  is  barely  keeping  up  with  evaluations. 

The  personnel  shortage  will  not  be  solved  near-term,  but  it  will  never  be  solved  if  a  pro¬ 
gram  is  not  undertaken  to  increase  the  expertise  pool.  The  degree  to  which  this  shortage 
of  trained  personnel  affects  operations  increases  in  severity  as  Options  2-5  are  consid¬ 
ered.  As  the  notional  estimates  in  Figure  8  illustrate,  Options  2-5  require  increasing 
numbers  of  not  only  people,  but  also  processes  and  technologies.  Requirements  for  Op¬ 
tion  6  are  in  addition  to  those  of  the  other  option  selected,  with  the  exception  of  Option  5 
from  which  the  Independent  Assessment  in  Option  6  picks  up  several  of  the  requirements 
of  Option  5.  Additionally,  the  alternate  assurance  methods  for  basic  security  software 
described  in  Options  4  and  5  may  reduce  overall  resource  requirements. 


Figure  8.  Rough  Order  of  Magnitude  (ROM)  Resource  Requirement 
8.8.2  Requirements 

The  safeguarding  of  the  cyberspace  requires  additional  funding.  The  growth  of  the 
evaluation  portion  of  the  NIAP  has  relegated  other  functions  to  a  back  burner.  NSA  is 
predominantly  doing  the  oversight  of  evaluations  by  themselves,  while  NIST  is  predomi¬ 
nantly  working  the  lab  certifications  through  NYLAP.  This  leaves  the  other  original 
functions  of  the  NIAP  unattended.  Each  option  carries  its  set  of  requirements  and  de¬ 
tailed  cost  estimates  should  be  obtained  through  the  NIAP  for  the  option  that  will  be  ex¬ 
ercised.  Even  if  funding  increases,  personnel  shortages  discussed  above  limit  the  solution 
in  the  near  term. 

Research 

The  topic  of  research  in  general  is  very  broad.  The  NIAP  should  pursue  research  that  is 
limited  to  evaluation  related  areas  such  as  metrics,  test  development,  composability,  as¬ 
surance  methods,  vulnerability  characterization,  and  tool  development  process.  However, 
under  options  3  to  5  this  would  include  a  general  awareness  of  other  research  in  the  area 
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of  cybersecurity  area.  The  independent  assessment  group  of  option  6  may  take  this  up,  if 
that  option  is  implemented. 

Education,  Training,  and  Awareness 

Should  be  for  the  benefit  of  all  stakeholders  and  the  general  consumer  of  IA  products  and 
not  limited  to  evaluation  personnel. 

Metrics  and  Tracking  Systems 

A  high  priority  should  be  placed  on  developing  the  metrics  that  will  provide  return  on 
investment  for  evaluated  products.  Once  identified  and  defined,  tracking  systems  must 
be  put  into  place  and  the  data  should  be  reported  at  least  annually  in  assessment  reports. 

Protection  Profile  Development 

It  is  not  intended  that  the  NIAP  develop  all  protection  profiles.  Rather  the  facilitation  of 
their  production  through  industry  consortia  and  academic  partners  should  be  pursued. 
The  current  facilitation  between  the  NIAP  and  NS  A  for  the  department  of  defense  is  a 
good  model  of  cooperation,  but  a  poor  model  of  where  responsibility  lies.  It  is  often  dif¬ 
ficult  for  the  consumer  to  separate  the  NS  A  effort  in  this  area  from  the  NIAP  itself. 

8.8.3  Security  Support  Group  (SSG) 

Options  2-5  all  recommend  establishing  a  SSG.  The  SSG  is  intended  to  be  a  part  of  the 
NIAP  structure.  The  tasking  of  this  SSG  varies  throughout  the  options  and  is  generally 
additive  up  to  Option  6,  where  the  group  providing  independent  assessments  may  sub¬ 
sume  the  responsibilities.  The  ISO  CC  standard  creates  a  global  market  for  secure  prod¬ 
ucts  because  of  mutual  recognition.  Some  of  the  above  recommendations  require 
changes  to  the  ISO  version  of  the  CC.  As  a  participant  in  the  international  standards  de¬ 
velopment  process  the  U.S.  cannot  dictate  changes  to  ISO  standards.  The  NIAP  (or  the 
independent  assessment  group  of  option  6)  should  decide  when  it  is  advisable  to  push  for 
changes  to  the  ISO  standard. 

8.8.4  Testing 

Stakeholder  expectations  revealed  several  NIAP  weaknesses.  For  example,  the  CC  testing 
of  software  security  functionality  and  vulnerabilities  do  not  become  nearly  detailed 
enough  until  levels  above  EAL4.  Most  stakeholders  prefer  or  anticipate  detailed  testing 
for  these  at  all  levels. 

8.8.5  Flaw  Remediation  and  Assurance  Maintenance 

While  stakeholders  (during  interviews)  did  not  discuss  flaw  remediation,  assurance  main¬ 
tenance  certainly  was  discussed.  The  literature  search  (as  detailed  in  Chapter  5)  provided 
for  flaw  remediation  as  an  expectation.  The  software  development  cycle  currently  con¬ 
sists  of  releases  and  fixes  and  possible  service  packs  which  are  rollups  of  fixes.  The  cur¬ 
rent  CC  has  provisions  for  both  flaw  remediation  and  maintenance  assurance,  but  they  are 
not  part  of  any  assurance  packages,  and  should  be  included  in  all  evaluations.  The  ab¬ 
sence  of  flaw  remediation  and  assurance  maintenance  makes  the  evaluation  certificate 
quickly  obsolete.  Moreover,  it  makes  the  integration  of  evaluations  with  C&A  efforts  in 
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Option  5  more  difficult,  because  the  assurance  of  subsequent  product  releases  is  not  ad¬ 
dressed.  Adding  these  requirements  to  PPs  or  acquisition  language  as  is  the  practice  in 
DoD,  is  good,  and  should  be  continued  until  all  evaluations  include  these  items. 

8.8.6  Formalization 

The  informal  nature  of  the  NIAP  places  it  at  a  decided  disadvantage  when  struggling  for 
prioritized  funds  and  negotiating  with  other  government  and  non-government  entities. 
Formalization  would  indicate  a  stronger  support  for  the  program  at  the  funding  level  and 
provide  it  a  stronger  basis  for  funding  priorities.  It  would  supply  a  sponsor,  requirements 
and  oversight.  It  would  also  provide  the  NIAP  a  stronger  position  in  the  international 
community.  This  lack  of  a  formal  charter  is  in  direct  contrast  to  the  Defense-wide  Infor¬ 
mation  Assurance  Program  (DIAP),  which  is  formally  chartered  by  the  DoD31. 

8.8.7  C&A  Interface 

A  number  of  suggestions  are  made  to  bolster  the  integration  between  product  evaluation 

and  the  process  of  C&A.  This  reduces  duplicative  effort  and  strengthens  the  overall  sys¬ 
tem  evaluation. 

8.8.8  Improvement  of  the  NIAP  Processes 

A  few  NIAP  process  improvements  are  in  the  roadmap  because  of  their  overall  impor¬ 
tance.  A  more  complete  list  of  the  NIAP  process  improvements  is  in  text  preceding  this 
analysis.  Specifically,  Chapter  5  suggests  numerous  improvements  to  the  NIAP  process. 

8.8.9  Consolidation 

Making  product  evaluation  more  universally  required  should  reduce  costs,  and  foster  an 
environment  of  greater  security.  However,  specific  steps  must  be  taken  before  requiring 
this  consolidation  (see  cost  reduction  below). 

8.8.10  Cost  Reduction 

Cost  reduction  refers  to  the  cost  of  evaluation.  The  NIAP  stakeholders  expressed  severe 
dissatisfaction  with  the  paperwork  burden  in  evaluating  products.  Likewise,  a  more  cost 
effective  method  of  providing  low  to  moderate  security  is  necessary  before  making  prod¬ 
uct  evaluation  more  universal  (Federal  government  wide  and  critical  infrastructure). 

8.8.11  Standards 

The  principle  interface  for  evaluations  is  the  CC  standard.  However,  a  number  of  related 
standards  need  to  be  developed.  The  specific  standards  need  to  be  detennined  in  option 
6;  however,  standardized  PPs,  Standardized  assurance  packages  (beyond  those  in  the 
CC),  and  security  Application  Protocol  Interfaces  (API)  are  probably  needed. 


31  See  Annex  D,  section  D.1.17  for  discussion  of  DIAP 
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8.9 


Other  Considerations 


8.9.1  Tradeoffs 

There  exists  a  tradeoff  between  the  optimism  for  alternative  paradigms  as  listed  in  Option 
6,  and  the  degree  to  which  funding  may  be  diverted  from  current  cybersecurity  evalua¬ 
tions  into  research.  Philosophically,  a  rapid  conversion  to  an  alternative  paradigm  would 
indicate  the  choice  of  Option  2  and  the  diversion  of  saved  funds  into  research  to  bring 
about  this  new  paradigm  as  quickly  as  possible.  Practically,  however,  no  new  paradigm 
has  been  identified  nor  are  any  approaching  maturation  at  this  time.  Breakthroughs  in  re¬ 
search  are  unpredictable.  Option  6  allows  for  the  gathering  of  data,  analysis,  and  a  heads- 
up  on  maturing  approaches.  This  will  provide  the  least  lead-time  to  affect  change. 

8.9.2  Centralized  Responsibility 

The  need  for  an  overarching  responsibility  of  the  security  of  the  nation’s  cyberspace  is  a 
real  one.  Currently  (as  delineated  in  the  text)  responsibility  for  pieces  of  this  problem 
runs  from  Department  of  Commerce  (DoC),  DoD,  DHS,  and  others  with  exceptions  for 
some  application  areas.  This  not  only  leads  to  the  potential  for  conflicts,  but  also  adds 
unnecessarily  to  complexity  and  duplication  of  effort.  Conflict  resolution  may  be  handled 
by  adjudication  to  higher  authority.  Complexity  and  duplication  of  effort  will  be  handled 
by  analysis  and  the  production  of  mandates  and  guidelines.  The  Independent  Assessment 
group  delineated  in  Option  6  is  intended  to  undertake  this  responsibility. 

8.9.3  Terminology 

Tenninology  is  currently  aimed  at  standards  developers,  evaluators,  and  oversight  per¬ 
sonnel.  The  term  assurance  was  not  well  understood  and  often  improperly  related  to 
strength  of  security.  For  example,  an  EAL2  product  might  be  harder  to  penetrate  than  an 
EAL  5  product,  yet  consumers  often  incorrectly  believe  the  opposite.  In  truth,  the  assur¬ 
ance  level  does  not  give  any  indication  of  the  difficulty  of  penetrating  of  the  product. 
The  assurance  level  primarily  describes  how  much  effort  and  evidence  examination  was 
expended  in  support  of  the  claims,  not  to  what  extent  the  product  provides  IA  functional¬ 
ity  and  strength.  The  paradigm  has  shifted  to  where  security  in  a  product  has  moved  from 
a  specialty  to  a  commodity  and  the  tenninology  should  follow  a  more  intuitive  and  com¬ 
mon  usage  path.  For  example,  products  should  be  rated  by:  home,  small  business,  finan¬ 
cial  industry,  commercial  business,  federal  government,  defense,  defense  critical,  intelli¬ 
gence,  etc.  Where  either  the  NIAP  or  the  Independent  Assessment  group  would  be  the 
keeper  of  the  definition  (and  definitions  are  developed  by  user  consortiums),  then  educa¬ 
tion  and  training  of  consumers  would  be  simplified  and  the  consumer  would  have  an  in¬ 
tuitive  feel  for  the  product’s  strengths.  These  definitions  might  embody  both  assurance 
and  strength  of  security.  While  the  recommendation  includes  a  shift  in  language  it  is  not 
suggested  to  remove  content  from  the  message,  and  the  complexities  of  the  tradeoffs  be¬ 
tween  risk  and  security  should  be  adequately  described,  in  order  to  not  dumb  down  the 
language  for  popular  consumption. 
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8.9.4 


Standardized  APIs 


The  advice  and  consent  described  under  Option  5  is  a  key  element  in  the  integration  of 
product  evaluations  with  C&A.  The  proper  setup  of  environments,  configuration  control 
and  the  development  of  “glue  logic”  are  all  key  to  reduced  vulnerabilities  in  software. 
The  advice  and  consent  will  provide  an  analysis  whereby  the  security  of  evaluated  prod¬ 
ucts  is  not  compromised  by  its  integration  into  a  system’s  environment.  “Glue  Logic”  is 
the  small  scripts,  and  programs  that  reformat  and  supplement/modify/combine  the  out¬ 
puts  of  program(s)  to  be  compatible  with  another  program’s  use.  In  complex  systems 
(that  use  many  interacting  products),  these  are  almost  always  needed.  A  concerted  pro¬ 
gram  of  standardization  of  APIs  for  security-enabled  software  can  reduce  their  use  (see 
section  8.7.11  on  standards). 

8.9.5  Requirements  Beyond  MRA 

Several  of  the  options  include  elements  that  extend  beyond  the  current  MRA;  for  exam¬ 
ple,  inclusion  of  maintenance  assurance  and  flaw  remediation,  and  requiring  the  use  of 
source  code  analysis  tools.  Maintenance  assurance  and  flaw  remediation  are  already  part 
of  the  CC,  but  evaluations  are  not  required  to  address  them.  Standards  for  analysis  tools 
would  have  to  be  established  before  their  use  could  be  mandated  under  an  MRA. 

If  additional  requirements  like  these  cannot  be  incorporated  under  the  MRA,  they  can  still 
be  added  as  US  evaluation  requirements.  NS  A  has  protection  profdes  that  require  vul¬ 
nerability  testing  (AVA_VLA3),  which  goes  beyond  EAL4  evaluations.  NSA  accepts 
evaluations  of  EAL4  products  under  the  MRA,  but  then  requires  additional  testing  by 
NSA  or  a  US  lab  to  satisfy  these  PPs. 

8.10  DoD  and  DHS  Recommended  Actions  to  prepare  for  the  roadmaps 

A  number  of  actions  by  DoD  and  DHS  are  recommended  to  achieve  the  most  useful  ap¬ 
proach  to  cybersecurity,  which  is  embodied  in  a  combination  of  options  5  and  6. 

1 .  It  is  recommended  that  DoD  continue  to: 

a.  Develop  Protection  Profiles  (PPs)  for  DoD  and  National  Security  applica¬ 
tions. 

b.  Require  conformance  to  PPs,  where  available. 

c.  Require  enhanced  vulnerability  testing  in  lower- level  EAL  packages. 

d.  Include  aspects  of  flaw  remediation  and  assurance  maintenance  in  PPs. 

2.  Additionally,  it  is  recommended  that  DoD: 

a.  Require  product  evaluations  to  include  maintenance  assurance  and  flaw 
remediation  packages  of  the  Common  Criteria. 

b.  Support  the  full  integration  of  product  evaluation  and  C&A  processes. 

c.  Support  the  development  and  use  of  software  tools  for  vulnerability  analy¬ 
ses. 
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d.  Participate  in  an  annual  assessment  and  review  of  the  nation’s  cybersecu¬ 
rity  posture. 

e.  Support  the  development  of  a  lower  cost,  alternate  form  of  assurance  for 
lower  assurance  products  (vulnerability  analysis  tools  needed). 

3.  It  is  recommended  that  DHS: 

a.  Support  vulnerability  testing  of  all  products  undergoing  evaluation. 

b.  Support  product  evaluations  to  include  maintenance  assurance  and  flaw 
remediation. 

c.  Support  the  full  integration  of  product  evaluation  and  C&A  processes. 

d.  Support  the  development  and  use  of  software  tools  for  vulnerability  analy¬ 
ses. 

e.  Support  the  development  of  a  lower  cost,  alternate  form  of  assurance  for 
lower  assurance  products. 

f.  Support  the  development  of  a  set  of  core  functionality  protection  profiles 
for  use  by  federal  departments  and  agencies,  by  critical  infrastructure 
components,  and  by  the  commercial  sector. 

g.  Support  the  use  of  core  protection  profiles,  where  applicable,  to  give 
product  buyers  confidence  in  the  product’s  security  functionality  and  suit¬ 
ability  for  use. 

h.  Support  the  full  integration  product  evaluation  and  C&A  processes  for 
federal  departments  and  agencies  and  critical  infrastructure  components 
(vulnerability  analysis  tools  and  a  lower  cost  alternative  form  of  assurance 
needed). 
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Annex  B  Acronyms 


AMA: 

AMP: 

ANSI: 

APNSA: 

ASD/NII: 

ASTM: 

ATE: 

AVA: 

C&A: 

CAE: 

CB: 

CC: 

CCA: 

CCEB: 

CCEL: 

CCEP: 

CCEVS: 

CCIMB: 

CCRA: 

CCTL: 

CEM: 

CEMEB: 

CERT/CC: 

CFR: 

Cl: 

CIA: 


A 

Assurance  Maintenance  Activity 

Assurance  Maintenance  Plan 

American  National  Standards  Institute 

Assistant  to  the  President  for  National  Security  Affairs 

Assistant  Secretary  of  Defense  Networking  and  Information  Integra¬ 
tion 

American  Society  of  Testing  Materials 
Assurance  Measures  for  Testing 
Assurance  Vulnerability  Assessment 

C 

(1)  Certification  and  accreditation  (2)  Certification  and  authorization 
to  operate  (FAA) 

Centers  of  Academic  Excellence 

Certification/validation  body 

(1)  Common  Criteria  (2)  Coordination  Center 

Clinger-Cohen  Act 

CC  Editing  Board 

Common  Criteria  Evaluation  Laboratory 
Commercial  COMSEC  Evaluation  Program 
Common  Criteria  Evaluation  and  Validation  Scheme 
CC  Implementation  Management  Board 
Common  Criteria  Recognition  Arrangement 
Common  Criteria  Testing  Laboratory 
Common  Evaluation  Methodology 
Common  Evaluation  Methodology  Editing  Board 
CERT*  Coordination  Center 
Code  of  Federal  Regulations 
Critical  Infrastructure 
Central  Intelligence  Agency 


B-l 


CIAO: 

(1)  Chief  Infrastructure  Assurance  Officer  (2)  Department  of  Com¬ 
merce  Critical  Infrastructure  Assurance  Office 

CIO: 

Chief  Information  Officer 

CIP: 

Critical  Infrastructure  Protection 

CLEF: 

Common  Criteria  Licensed  Evaluation  Facility 

CM: 

(1)  Common  Methodology  (2)  Configuration  management 

CMP: 

Certificate  Maintenance  Program 

CMR: 

Certificate  Maintenance  Report 

CMSR: 

Certificate  Maintenance  Summary  Report 

CMT  LAP: 

NVLAP®  Cryptographic  Module  Testing  Laboratory  Accreditation 
Program 

CMV: 

Cryptographic  Module  Validation 

CMVP: 

Cryptographic  Module  Validation  Program 

CND: 

Computer  Network  Defense 

CNDS: 

Computer  Network  Defense  Services 

CNSS: 

Committee  on  National  Security  Systems 

COMPUSEC: 

COMPUter  SECurity 

COMSEC: 

COMmunications  SECurity 

COTS: 

Commercial  “off  the  shelf’ 

CRS: 

Congressional  Research  Service 

CSA: 

Computer  Security  Act 

CSEC: 

Computer  Security  Evaluation  Center 

CSI: 

Computer  Security  Institute 

CSIA: 

Cyber  Security  Industry  Alliance 

CSD: 

Computer  Security  Division 

CSTT: 

Cryptographic  Support  Test  Tool 

CTSC: 

Chenega  Technology  Services  Corporation 

D 

(1)  Designated  Accrediting  Authority  (2)  Designated  Approving  Au¬ 
thority 

DAA: 

DCA: 

Deferred  Compliance  Authorization 

DCI: 

Director  of  Central  Intelligence 

DCID: 

Director  of  Central  Intelligence  Directives 
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DDCI/CM: 

Deputy  Director  of  Central  Intelligence  for  Community  Management 

DEA: 

Drug  Enforcement  Agency 

DHHS: 

Department  of  Health  and  Human  Services 

DHS: 

Department  of  Homeland  Security 

DIA: 

Defense  Intelligence  Agency 

DISA: 

Defense  Information  Systems  Agency 

DIACAP: 

Defense  Information  Assurance  Certification  and  Accreditation  Proc¬ 
ess 

DIRNSA: 

Director  National  Security  Agency 

DITSCAP: 

DoD  Information  Technology  Security  and  Certification  Process 

DLA: 

Defense  Logistics  Agency 

DOC: 

Department  of  Commerce 

DoD: 

Department  of  Defense.  Also  used  as  the  tag  for  stakeholder  class  of 
DoD  users. 

DoDD: 

DoD  Directive 

DoDI: 

DoD  Instruction 

DSB: 

Defense  Science  Board 

DTRA: 

Defense  Threat  Reduction  Agency 

DVA: 

Department  of  Veteran  Affairs 

E 

E-Government  Act 

E-Gov: 

EAL: 

Evaluation  Assurance  Level 

EAP: 

Evaluation  Acceptance  Package 

EF: 

Evaluation  facility,  an  organization  that  carries  out  evaluations  inde¬ 
pendently  of  the  developers  of  the  IT  products  or  protection  profiles, 
usually  on  a  commercial  basis. 

EMSEC: 

EMissions  SECurity 

EO: 

Executive  Order 

EOP: 

Executive  Office  of  the  President 

ESR: 

Evaluation  Summary  Report 

ET&A: 

Education,  Training,  and  Awareness 

ETR: 

Evaluation  Technical  Report 

T 7 

FA  A: 

r 

Federal  Aviation  Administration 
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FBI: 

FCC: 

FedNonDoD: 

FEMA: 

FFRDC: 

FIPS: 

FIRMR: 

FISMA: 

FRS: 

FTC: 

GAO: 

GIG: 

GISR: 

GLB: 

GOTS: 

GSA: 

HASC: 

HIPPA: 

HPSCI: 

HAS: 

HSPD: 

IA: 

IATF: 

IATFF : 

I  AW: 

IC: 

IDA: 

IEC: 

IEEE: 


Federal  Bureau  of  Investigation 
Federal  Communications  Commission 

Used  as  the  tag  for  stakeholder  class  of  Federal  users  that  is  not  DoD. 

Federal  Emergency  Management  Agency 

Federally-Funded  Research  and  Development  Center 

Federal  Infonnation  Processing  Standard 

Federal  Information  Resources  Management  Regulations 

Federal  Information  Security  Management  Act  of  2002 

Federal  Reserve  System 

Federal  Trade  Commission 

G 

Government  Accountability  Office  (prior  to  July  7,  2004,  the  General 
Accounting  Office) 

Global  Infonnation  Grid 

Government  Information  Security  Reform 

Gramm-Leach-Bliley  Act 

Government  Off  the  Shelf 

Government  Services  Administration 

H 

House  Armed  Services  Committee 

Health  Insurance  Portability  and  Accountability  Act  of  1996 
House  Permanent  Select  Committee  on  Intelligence 
Homeland  Security  Act 
Homeland  Security  Presidential  Directive 

I 

Information  Assurance 

Information  Assurance  Technical  Framework 

Information  Assurance  Technical  Framework  Forum 

Indications  and  Warnings 

Intelligence  Community 

Institute  for  Defense  Analyses 

International  Electro-technical  Commission 

Institute  of  Electrical  and  Electronics  Engineers 


B-4 


IETF: 

IG: 

INFOSEC: 

INS: 

IRA: 

IRTPA: 

ISPAB: 

ISO: 

ISS: 

IT: 

ITMRA: 

ITSEC: 

ITSEF: 

ITSEM: 

ITU: 

JCS: 

KPA: 

MR: 

MRA: 

MSR: 

NASA: 

NCS: 

NDAA: 

NDI: 

NERC: 

NGA: 

NIACAP: 

NIAP: 


Internet  Engineering  Task  Force 
Inspectors  General 
INFOrmation  SECurity 
Immigration  and  Naturalization  Service 
Intelligence  Reform  Act 

Intelligence  Reform  and  Terrorism  Prevent  Act  2004 
Information  Security  and  Privacy  Advisory  Board 
International  Organization  for  Standards 
Information  Systems  Security 
Information  Technology 

Information  Technology  Management  Reform  Act  of  1996 
Information  Technology  Security  Evaluation  Criteria 
IT  Security  Evaluation  Facility 
Information  Technology  Security  Evaluation  Manual 
International  Telecommunications  Union 

J 

Joint  Chiefs  of  Staff 

K 

Key  Process  Area 

M 

(1)  Memorandum  for  Record  (2)  Management  Representative 
Mutual  Recognition  Arrangement 
Monthly  Summary  Report 

N 

National  Aeronautics  and  Space  Administration 

National  Communications  System 

National  Defense  Authorization  Act 

Non  Developmental  Item 

North  American  Electric  Reliability  Council 

National  Geospatial-Intelligence  Agency 

National  Infonnation  Assurance  Certification  and  Accreditation  Proc¬ 
ess 

National  Infonnation  Assurance  Partnership 
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NIPC: 

NIST: 

NRIC: 

NRC: 

NRO: 

NSA: 

NSD: 

NSF: 

NSI: 

NSS: 

NSTISSAM: 

NSTISSC: 

NSTISSI: 

NSTISSP: 

NTTAA: 

NVLAP: 

OD: 

ODRB: 

OMB: 

OPM: 

OPSEC: 

OR: 

OSP: 

OSTP: 

PCAOB: 

PDD: 

POA&M: 

POC: 


National  Infrastructure  Protection  Center 

National  Institute  of  Standards  and  Technology 

Network  Reliability  and  Interoperability  Council  of  the  FCC 

Nuclear  Regulatory  Commission 

National  Reconnaissance  Office 

National  Security  Agency;  National  Security  Act 

National  Security  Directive 

National  Science  Foundation 

National  Security  Infonnation 

National  Security  Systems 

National  Security  Telecommunications  and  Information  Systems  Secu 
rity  Advisory/Information  Memorandum 

National  Security  Telecommunications  and  Information  Systems  Secu 
rity  Committee  (U.S.) 

National  Security  Telecommunications  and  Information  Systems  Secu 
rity  Instruction 

National  Security  Telecommunications  and  Information  Systems  Secu 
rity  Policy 

National  Technology  Transfer  and  Advancement  Act  of  1995 
National  Voluntary  Laboratory  Accreditation  Program 

O 

Observation  Decision 
Observation  Decision  Review  Board 
Office  of  Management  and  Budget 
Office  of  Personnel  Management 
Operations  SECurity 
Observation  Report 
Organizational  Security  Policy 

White  House  Office  of  Science  and  Technology  Policy 

P 

Public  Company  Accounting  Oversight  Board 
Presidential  Decision  Directive 
Plan  of  Action  and  Milestones 
Point  of  Contact 
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POSIX: 

Portable  Operating  System  Interface 

PP: 

Protection  Profile 

PRA: 

Paperwork  Reduction  Act 

R 

RA: 

Registration  Authority 

RCR: 

Representation  CoRrespondence 

RI: 

Request  for  Interpretation 

ROI: 

Return  on  Investment 

S 

SAR: 

Security  assurance  requirement 

SASC: 

Senate  Armed  Services  Committee 

SEC: 

Securities  and  Exchange  Commission 

SEI: 

Software  Engineering  Institute 

SF: 

Security  Function 

SFP: 

Security  Function  Policy 

SFR: 

Security  Functional  Requirement 

S/L/T: 

State/Focal/Tribal 

SOF: 

Strength  Of  Function 

SOX: 

Sarbanes-Oxley  Act 

SP: 

Special  Publication 

SSA: 

Sector  Specific  Agency 

SSAA: 

System  Security  Authorization  Agreement 

SSCI: 

Senate  Select  Committee  on  Intelligence 

SSE-CMM: 

System  Security  Engineering  Capability  Maturity  Model 

SSG: 

Security  Support  Group 

SSP: 

Scientific  Subroutine  Package 

ST: 

Security  Target 

ST&E: 

Security  Test  and  Evaluation 

T 

TCSEC: 

Trusted  Computer  System  Evaluation  Criteria 

TOE: 

Target  of  Evaluation 

TPEP: 

Trusted  Product  Evaluation  Program 

TSC: 

TSF  Scope  of  Control 
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TSF: 

TOE  Security  Functions 

TSFI: 

TSF  Interface 

TSP: 

TOE  Security  Policy 

TSS: 

TOE  Summary  Specification 

TTAP: 

Trust  Technology  Assessment  Program 

TTP: 

Trusted  Third  Party 

U 

UL: 

Underwriters’  Laboratory 

USG: 

U.S.  Government 

V 

VA: 

Veterans  Administration 

VID: 

Validation  Identification  Number 

VPL: 

Validated  Products  List 

VR: 

Validation  Report 
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Annex  C  Glossary 

C.l  General  Terminology 

A 

Acceptance  Phase:  Start  of  an  assurance  maintenance  cycle  in  which  the  developer  es¬ 
tablishes  plans  and  procedures  for  assurance  maintenance  that  are  independently 
validated  by  an  evaluator. 

Accreditation:  (1)  Formal  recognition  that  a  laboratory  is  competent  to  carry  out  spe¬ 
cific  tests  or  calibration  or  types  of  tests  or  calibrations.  (2)  Confirmation  by  an  ac¬ 
creditation  body  as  meeting  a  predetermined  standard  of  impartiality  and  general 
technical,  methodological,  and  procedural  competence,  (see  section  C.2  for  context 
associated  with  Certification  and  Accreditation  [definition  2]) 

Accreditation  Body:  An  independent  organization  responsible  for  assessing  the  per¬ 
formance  of  other  organizations  against  a  recognized  standard,  and  for  formally 
confirming  the  status  of  those  that  meet  the  standard. 

Accredited:  Formally  confirmed  by  an  accreditation  body  as  meeting  a  predetermined 
standard  of  impartiality  and  general  technical,  methodological,  and  procedural 
competence. 

Action:  Explicitly  described  CC  evaluator  action  element  or  one  derived  from  a  speci¬ 
fied  developer  action  element. 

Activity:  Application  of  a  CC  assurance  class. 

A.NOEVIL:  Assumption  that  authorized  administrators  is  non-hostile  and  follows  all 
administrator  guidance;  however,  they  are  capable  of  error. 

Applicant:  Entity  (organization,  individual,  etc.)  requesting  the  assignment  of  a  register 
entry  and  entry  label. 

Approval  Policy:  A  part  of  the  essential  documentation  of  the  Common  Criteria  Evalua¬ 
tion  and  Validation  Scheme,  setting  out  the  procedures  for  making  an  application  to 
be  approved  as  a  CCTL  and  placed  on  the  NIAP  Approved  Laboratories  List  and  for 
the  processing  of  such  applications  and  of  the  requirements  which  an  applicant  must 
fulfill  in  order  to  qualify. 

Approved  Lab  List:  The  list  of  approved  CCTLs  authorized  by  the  NIAP  Validation 
Body  to  conduct  IT  security  evaluations  within  the  Common  Criteria  Evaluation 
and  Validation  Scheme. 

Approved  Test  Method  List:  The  list  of  approved  test  methods  maintained  by  the  NIAP 
Validation  Body,  which  can  be  selected  by  a  CCTL  in  choosing  its  scope  of  accredi¬ 
tation,  i.e.,  the  types  of  IT  security  evaluations  that  it  will  be  authorized  to  conduct 
using  NIAP-approved  test  methods. 

Approved:  Assessment  by  a  national  evaluation  body  as  being  technically  competent  in 
the  specific  field  of  IT  security  evaluation  and  formally  authorized  to  carry  out 
evaluations  within  the  context  of  the  CCEVS. 
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Assets:  (1)  Information  or  resources  to  be  protected  by  the  countenneasures  of  a  TOE; 
assets  may  be  external  to  the  TOE  but  within  the  IT  environment.  (2)  Anything  that 
has  value  to  the  organization. 

Assignment:  Specification  of  a  parameter  filled  in  when  an  element  is  used  in  a  Protec¬ 
tion  Profile  (PP)  or  Security  Target  (ST). 

Assumption:  Security  aspects  of  the  environment  in  which  the  TOE  will  or  is  intended 
to  be  used. 

Assurance:  Grounds  for  confidence  that  an  entity  meets  its  security  objectives. 

Assurance  Maintenance  Plan:  Part  of  the  formal  assurance  maintenance  documentation 
submitted  to  the  validation  body  by  the  sponsor  of  an  evaluation  that  identifies  the 
plans  and  procedures  that  a  developer  is  to  implement  in  order  to  ensure  that  the  as¬ 
surance  that  was  established  in  the  certified/validated  TOE  is  maintained  as  changes 
are  made  to  the  target  of  evaluation  (TOE)  or  its  environment. 

Attack  Potential:  Perceived  potential  for  success  of  an  attack,  should  an  attack  be 
launched,  expressed  in  terms  of  an  attacker’s  expertise,  resources,  and  motivation. 

Augmented:  Addition  of  one  or  more  assurance  components  from  Part  3  of  the  CC  to  an 
EAL  that  is  not  normally  part  of  that  EAL. 

Authenticity:  Property  that  ensures  that  the  identity  of  a  subject  or  resource  is  the  one 
claimed.  Authenticity  applies  to  entities  such  as  users,  processes,  systems,  and  in¬ 
formation. 

Authorized  User:  User  who  may,  in  accordance  with  the  TOE  security  policy  (TSP), 
perfonn  an  operation. 

Availability:  (1)  Property  of  being  accessible  and  usable  upon  demand  by  an  authorized 
entity.  (2)  Prevention  of  unauthorized  withholding  of  information  resources. 

B 

Baseline  Controls:  A  minimum  set  of  safeguards  established  for  a  system  or  organiza¬ 
tion. 

Best  Practices:  processes,  practices,  and  systems  identified  in  public  and  private  organi¬ 
zations  that  performed  exceptionally  well  and  are  widely  recognized  as  improving 
an  organization's  performance  and  efficiency  in  specific  areas.  Successfully  identi¬ 
fying  and  applying  best  practices  can  reduce  business  expenses  and  improve  organ¬ 
izational  efficiency. 

C 

Certification:  (1)  the  procedure  by  which  a  third  party  gives  written  assurance  that  a 
product,  process,  or  service  conforms  to  specified  requirements  or  standards.  (2)  a 
comprehensive  assessment  of  the  management,  operational,  and  technical  security 
controls  in  an  information  system,  made  in  support  of  security  accreditation,  to  de¬ 
termine  the  extent  to  which  the  controls  are  implemented  correctly,  operating  as  in¬ 
tended,  and  producing  the  desired  outcome  with  respect  to  meeting  the  security  re¬ 
quirements  for  the  system. 
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Certification/validation  body:  an  organization  responsible  for  carrying  out  certifica¬ 
tion/validation  and  for  overseeing  the  day-to-day  operation  of  an  evaluation  and 
certification  or  validation  scheme. 

Certificate  Authorizing  Participant:  National  Evaluation  Authority  and  CCRA  signa¬ 
tory  that  issues  CC  Certificates  and  recognizes  those  issued  by  other  National 
Evaluation  Authorities. 

Certificate  Consuming  Participant:  National  Evaluation  Authority  and  CCRA  signa¬ 
tory  that  recognizes  CC  Certificates  issued  by  other  National  Evaluation  Authorities 
but  at  present  do  not  issue  any  certificates  itself. 

Certificate  of  Accreditation:  Document  issued  by  the  National  Voluntary  Laboratory 
Accreditation  Program  (NVLAP®)  or  other  national  evaluation  authority  to  a  labo¬ 
ratory  that  has  met  the  criteria  and  conditions  for  accreditation.  A  current  Certificate 
of  Accreditation  may  be  used  as  proof  of  accredited  status  and  is  always  accompa¬ 
nied  by  a  Scope  of  Accreditation. 

Certificate  Maintenance  Program:  a  program  within  the  CCEVS  that  allows  a  sponsor 
to  maintain  a  CC  Certificate  by  providing  a  means  to  ensure  that  a  validated  TOE 
will  continue  to  meet  its  Security  Target  as  changes  are  made  to  the  IT  product  or  its 
environment. 

Certificate  Maintenance  Report:  a  report  prepared  by  a  CCTL  for  the  evaluation  au¬ 
thority  detailing  the  results  of  their  evaluation  maintenance  activities  conducted  on 
behalf  of  a  sponsor. 

Certificate  Maintenance  Summary  Report:  an  annual  report  prepared  by  a  sponsor  for 
the  evaluation  authority  providing  a  summary  of  all  certificate  maintenance  activi¬ 
ties  conducted  during  the  previous  year. 

Certification/Validation:  (1)  Process  carried  out  by  a  CB  leading  to  the  issuance  of  a 
CC  Certificate;  (2)  comprehensive  evaluation  of  the  technical  and  non-technical  se¬ 
curity  features  of  an  IT  system  and  other  safeguards,  made  in  support  of  the  accredi¬ 
tation  process,  to  establish  the  extent  to  which  a  particular  design  and  implementa¬ 
tion  meets  a  set  of  specified  security  requirements,  (see  section  C.2  for  context  in¬ 
formation) 

Certification/Validation  Report:  Public  document  issued  by  a  CB  that  summarizes  the 
results  of  an  evaluation  and  confirms  the  overall  results — that  is,  the  evaluation  has 
been  properly  carried  out;  the  evaluation  criteria,  evaluation  methods,  and  other 
procedures  have  been  correctly  applied;  and  the  conclusions  of  the  Evaluation 
Technical  Report  are  consistent  with  evidence  adduced. 

CC  Certificate:  A  brief  public  document  issued  by  the  NIAP  Validation  Body  under  the 
authority  of  NIST  and  NSA  which  confirms  that  an  IT  product  or  protection  profile 
has  successfully  completed  evaluation  by  a  CCTL.  A  Common  Criteria  certificate 
always  has  associated  with  it,  a  validation  report. 

Certified  TOE:  (1)  Product  or  system  and  its  associated  guidance  that,  having  been  a 
TOE  under  evaluation,  has  completed  the  evaluation,  its  ST,  certification  report,  and 
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certificate  having  been  published.  (2)  Version  of  TOE  that  was  evaluated,  awarded  a 
CC  Certificate,  and  is  listed  in  an  evaluation  authority’s  Evaluated  Products  List. 

Certified/Validated  Products  List:  Public  document  that  summarizes  and  confirms  the 
results  of  an  evaluation  and  lists  current  valid  CC  Certificates  in  accordance  with 
the  CCRA. 

Check:  Similar  to,  but  less  rigorous  than,  confirm  or  verify;  a  quick  detennination  to  be 
made  by  the  evaluator,  perhaps  requiring  only  a  cursory  analysis,  or  perhaps  no 
analysis  at  all. 

Class:  Grouping  of  security  requirements  that  share  a  common  focus;  members  of  a  class 
are  termed  families. 

Coherent:  Entity  that  is  logically  ordered  and  has  a  discernible  meaning;  for  documenta¬ 
tion,  this  adjective  addresses  both  the  actual  text  and  the  structure  of  the  document, 
in  tenns  of  whether  it  is  understandable  by  its  target  audience. 

Common  Criteria:  Common  Criteria  for  Information  Technology  Security  Evaluation, 
the  title  of  a  set  of  documents  describing  a  particular  set  of  IT  security  evaluation 
criteria. 

Common  Criteria  Certificate:  (1)  Public  document  issued  by  a  compliant  CB  and  au¬ 
thorized  by  a  participant  that  confirms  that  a  specific  IT  product  or  Protection  Pro¬ 
file  has  successfully  completed  evaluation  by  an  IT  security  evaluation  facility 
(ITSEF);  a  CC  Certificate  always  has  associated  with  it  a  certification  and  valida¬ 
tion  report.  (2)  Fonnal  recognition  by  the  NIAP®  validation  body  that  the  IT  secu¬ 
rity  evaluation  has  been  conducted  in  accordance  with  the  CCEVS  requirements  us¬ 
ing  the  CC  and  CM.  A  product  that  has  received  a  CC  Certificate  is  placed  on 
NIAP®’s  Validated  Products  List. 

Common  Criteria  Evaluation  and  Validation  Scheme:  The  program  developed  by 
NIST  and  NSA  as  part  of  the  National  Information  Assurance  Partnership  (NIAP) 
establishing  an  organizational  and  technical  framework  to  evaluate  the  trustworthi¬ 
ness  of  IT  Products  and  protection  profiles. 

Common  Criteria  Implementation  Management  Board:  conducted  trial  evaluations  of 
first  draft  of  CC  and  developed  second  draft  of  CC. 

Common  Criteria  Interpretation  Management  Board:  renders  CC  interpretations  to 
facilitate  consistent  evaluation  results  under  the  Common  Criteria  Recognition 
Agreement  (CCRA). 

Common  Criteria  Testing  Laboratory:  Within  the  context  of  the  Common  Criteria 
Evaluation  and  Validation  Scheme  (CCEVS),  an  IT  security  evaluation  facility,  ac¬ 
credited  by  the  National  Voluntary  Laboratory  Accreditation  Program  (NVLAP) 
and  approved  by  the  NIAP  Validation  Body  to  conduct  Common  Criteria-based 
evaluations. 

Common  Evaluation  Methodology:  Common  Methodology  for  Information  Technol¬ 
ogy  Security  Evaluations  -  a  technical  document  that  describes  a  set  of  IT  security 
evaluation  methods. 
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Common  Evaluation  Methodology  Editing  Board:  CCRA  participants  involved  in  de¬ 
velopment  of  CEM. 

Common  Methodology  for  Information  Technology  Security  Evaluation:  a  technical 
document  that  describes  a  particular  set  of  IT  security  evaluation  methods,  also  re¬ 
ferred  to  as  CEM. 

Communications  Security:  measures  and  controls  taken  to  deny  unauthorized  persons 
information  derived  from  telecommunications  and  to  ensure  the  authenticity  of  such 
telecommunications.  COMSEC  includes  cryptosecurity,  transmission  security, 
emissions  security,  and  physical  security  of  COMSEC  material. 

Complete:  All  necessary  parts  of  an  entity  have  been  provided.  In  terms  of  documenta¬ 
tion,  this  means  that  all  relevant  information  is  covered  in  the  documentation,  at 
such  a  level  of  detail  that  no  further  explanation  is  required  at  that  level  of  abstrac¬ 
tion. 

Component  TOE:  TOE  that  fonns  part  of  a  composite  TOE;  the  lowest  level  TOE  in  an 
IT  product  or  system. 

Components:  Specific  set  of  security  requirements  that  are  constructed  from  elements; 
the  smallest  selectable  set  of  elements  that  may  be  included  in  a  PP,  an  ST,  or  a 
package. 

Composability:  Mathematical  problem  where  several  evaluated  products  are  used  to 
make  up  a  system.  Their  security  features  and  metrics  and  the  way  they  are  com¬ 
bined  are  then  used  to  compute  a  system  security  set  of  metrics.  The  problem  is  not 
yet  solved. 

Composite  TOE:  TOE  composed  of  multiple  component  TOEs;  the  highest  level  TOE 
in  an  IT  product  or  system. 

Computer  Security:  (1)  preventing,  detecting,  and  minimizing  the  consequences  of  un¬ 
authorized  actions  by  users  (authorized  and  unauthorized)  of  a  computer  system.  (2) 
measures  and  controls  that  ensure  confidentiality,  integrity,  and  availability  of  in¬ 
formation  system  assets  including  hardware,  software,  firmware,  and  information 
being  processed,  stored,  and  communicated. 

Confidentiality:  The  prevention  of  unauthorized  disclosure  of  information. 

Confirm:  To  review  in  detail  in  order  to  make  an  independent  determination  of  suffi¬ 
ciency,  with  the  level  of  rigor  required  depending  on  the  nature  of  the  subject  mat¬ 
ter;  applicable  to  evaluator  actions. 

Connectivity:  Property  of  the  TOE  that  allows  interaction  with  IT  entities  external  to  the 
TOE.  This  includes  exchange  of  data  by  wire  or  by  wireless  means,  over  any  dis¬ 
tance  in  any  environment  or  configuration. 

Consistent:  Relationship  between  two  or  more  entities,  indicating  that  there  are  no  ap¬ 
parent  contradictions  between  these  entities. 

Corrective  security  objective:  Security  objectives  that  require  the  TOE  to  take  action  in 
response  to  potential  security  violations  or  other  undesirable  events,  in  order  to  pre¬ 
serve  or  return  to  a  secure  state  and/or  limit  any  damage  caused. 
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Counter:  Offset,  nullify,  defensive  response  (i.e.,  a  security  objective  that  mitigates  a 
particular  threat  but  does  not  necessarily  indicate  that  the  threat  is  completely  eradi¬ 
cated  as  a  result). 

Critical  Infrastructure  Protection:  banking  and  finance,  energy,  chemical  sites, 
transportation,  telecommunications,  Government  facilities,  dams,  national 
monuments  and  icons,  cybersecurity  is  a  key  element  of  infrastructure  protection. 

Cybersecurity:  the  prevention  of  damage  to,  the  protection  of,  and  the  restoration  of 
computers,  electronic  communications  systems,  electronic  communication  services, 
wire  communications,  and  electronic  communications,  including  infonnation  con¬ 
tained  therein,  to  ensure  its  availability,  integrity,  authentication,  confidentiality, 
and  nonrepudiation,  (see  section  C.2  for  context  of  this  and  related  terms). 

Cyberterrorism:  a  criminal  act  perpetrated  through  computers  resulting  in  violence, 
death  and/or  destruction,  and  creating  terror  for  the  purpose  of  coercing  a  govern¬ 
ment  to  change  its  policies. 

Cryptographic  Algorithm  Testing:  Input/output  testing  to  determine  whether  the  im¬ 
plementation  conforms  to  the  specification. 

Cryptographic  Boundary:  Explicitly  defined  contiguous  perimeter  that  establishes  the 
physical  bounds  of  a  cryptographic  module 

Cryptographic  Module:  Set  of  hardware,  software,  firmware,  or  a  combination  thereof 
that  implements  cryptographic  logic  or  processes,  including  cryptographic  algo¬ 
rithms  and  key  generation,  and  is  contained  within  the  cryptographic  boundary  of 
the  module. 

Cryptographic  Module  Validation:  the  act  of  determining  if  a  cryptographic  module 
conforms  to  the  requirements  of  FIPS  PUB  140-2. 

Cryptographic  Module  Validation  Program:  a  program  run  jointly  by  the  Communica¬ 
tions  Security  Establishment  (CSE)  of  the  Government  of  Canada  and  the  National 
Institutes  of  Standards  and  Technology  (NIST)  that  focuses  on  security  confor¬ 
mance  testing  of  a  cryptographic  module  against  FIPS  PUB  140-2,  Security  Re¬ 
quirements  for  Cryptographic  Modules,  and  other  related  cryptographic  standards. 

Cryptographic  Support  Test  Tool:  used  as  part  of  Cryptographic  Module  Validation 
Program  (CMVP). 

Current  version  of  TOE:  Version  of  TOE  that  differs  in  some  respect  from  the  certified 
version,  such  as  (1)  a  new  release  of  the  TOE,  (2)  a  certified  version  with  patches  to 
correct  subsequently  discovered  bugs,  and  (3)  the  same  basic  version  of  the  TOE 
but  on  a  different  hardware  or  software  platform. 

D 

Data  Integrity:  Property  that  data  has  not  been  altered  or  destroyed  in  an  unauthorized 
manner. 

Demonstrate:  Analysis  leading  to  a  conclusion;  less  rigorous  than  a  proof. 
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Dependency:  Relationship  between  requirements  such  that  the  requirement  that  is  de¬ 
pended  upon  must  normally  be  satisfied  for  the  other  requirements  to  be  able  to 
meet  their  objectives. 

Depth:  Level  of  design  and  implementation  that  is  being  evaluated. 

Describe:  Provide  specific  details  about  an  entity. 

Detective  Security  Objective:  Security  objectives  that  provide  the  means  to  detect  and 
monitor  the  occurrence  of  events  relevant  to  the  secure  operation  of  the  TOE. 

Determine:  Conducting  an  independent  analysis,  usually  in  the  absence  of  any  previous 
analysis  having  been  performed,  with  the  objective  of  reaching  a  particular  conclu¬ 
sion;  differs  from  confirm  or  verify,  as  these  terms  imply  that  an  analysis  has  al¬ 
ready  been  performed  that  must  be  reviewed. 

E 

Evaluation  Facility:  an  organization  that  carries  out  evaluations  independently  of  the 
developers  of  the  IT  products  or  protection  profiles,  usually  on  a  commercial  basis. 

Element:  Indivisible  security  requirement  that  can  be  verified  by  the  evaluation;  lowest 
level  security  requirement  from  which  components  are  constructed. 

Emissions  security:  protection  resulting  from  measures  taken  to  deny  unauthorized  per¬ 
sons  infonnation  derived  from  the  interception  and  analysis  of  compromising  ema¬ 
nations  from  crypto  equipment  or  IT  systems. 

Entry  Label:  Naming  information  that  uniquely  identifies  a  registered  PP  or  package. 

Evaluation:  The  assessment  of  an  IT  product  against  the  Common  Criteria  using  the 
Common  Evaluation  Methodology  to  determine  whether  or  not  the  claims  made  are 
justified;  or  the  assessment  of  a  protection  profile  against  the  Common  Criteria  us¬ 
ing  the  Common  Evaluation  Methodology  to  determine  if  the  profile  is  complete, 
consistent,  technically  sound  and  hence  suitable  for  use  as  a  statement  of  require¬ 
ments  for  one  or  more  TOEs  that  may  be  evaluated. 

Evaluation  Acceptance  Package:  A  set  of  documentation  from  the  CCTL  consisting  of 
a  complete  security  target  for  the  Target  of  Evaluation  (TOE)  and  a  complete 
evaluation  work  plan  detailing  the  inputs,  actions  and  timelines  for  the  conduct  of 
the  evaluation;  and  the  identification  of  points  of  contact  for  both  the  CCTL  and  the 
sponsor  of  the  evaluation. 

Evaluation  Authority:  National  body  that  implements  the  CC  for  a  specific  community 
by  means  of  an  evaluation  scheme  and  thereby  sets  the  standards  and  monitors  the 
quality  of  evaluations  conducted  by  CBs  within  that  community. 

Evaluation  Scheme:  See  Common  Criteria  Evaluation  and  Validation  Scheme. 

Evaluation  Summary  Report:  a  report  issued  by  an  overseer  and  submitted  to  an 
evaluation  authority  that  documents  the  oversight  verdict  and  its  justification. 

Evaluation  Technical  Report:  A  report  giving  the  details  of  the  findings  of  an  evalua¬ 
tion,  submitted  by  the  CCTL  to  the  CCEVS  Validation  Body  as  the  principal  basis 
for  the  validation  report. 
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Evaluation  Work  Plan:  A  document  produced  by  a  CCTL  detailing  the  organization, 
schedule,  and  planned  activities  for  an  IT  security  evaluation. 

Evaluator  Action  Element:  Assurance  requirement  stated  in  Part  3  of  the  CC  that  repre¬ 
sents  a  TOE  evaluator’s  responsibilities  in  verifying  the  security  claims  made  in  the 
Security  Target  of  a  TOE. 

Exhaustive:  Used  to  describe  the  conduct  of  an  analysis  or  other  activity;  related  to  sys¬ 
tematic  but  considerably  stronger  in  that  it  indicates  not  only  that  a  methodical  ap¬ 
proach  has  been  taken  to  perform  the  analysis  or  activity  according  to  an  unambi¬ 
guous  plan  but  also  that  the  plan  followed  is  sufficient  to  ensure  that  all  possible 
avenues  have  been  exercised. 

Explicit  Requirements:  Functional  security  requirements  or  security  assurance  require¬ 
ments  specified  in  a  PP  or  ST  that  satisfy  a  specific  consumer  need  but  do  not  origi¬ 
nate  from  the  CC  catalog  of  standardized  components  (see  also  Refinement  and  Ex¬ 
tended). 

Extended:  Addition  to  an  ST  or  PP  of  requirements  not  contained  in  Part  2  or  assurance 
requirements  not  contained  in  Part  3  of  the  CC;  extensibility  (see  also  Explicit  re¬ 
quirements  and  Refinement). 

External  IT  Entity:  Any  IT  product  or  system,  distrusted,  or  trusted,  outside  of  the  TOE 
that  interacts  with  the  TOE. 


F 

Family:  Grouping  of  security  requirements  that  share  security  objectives  but  may  differ 
in  emphasis  or  rigor;  the  members  of  a  family  are  termed  components. 

Federally-Funded  Research  and  Development  Centers:  Members  of  various  FFRDCs 
are  used  as  validators  in  the  U.S.  scheme. 

Formal:  Expressed  in  a  restricted  syntax  language  with  defined  semantics  based  on 
well-established  mathematical  concepts. 

G 

Governance:  Used  as  the  tag  for  stakeholder  class  of  individuals  who  help  develop 
guidance  and  policy  over  IA  relevant  software. 

H 

Hierarchy:  Ordering  of  components  within  a  family  to  represent  increasing  strength  or 
capability  of  security  requirements  that  share  a  common  purpose;  on  occasion,  par¬ 
tial  ordering  is  used  to  illustrate  the  relationship  between  nonhierarchical  sets. 

I 

Information  Assurance:  Conducting  those  operations  that  protect  and  defend  informa¬ 
tion  and  information  systems  by  ensuring  availability,  integrity,  authentication,  con¬ 
fidentiality,  and  non-repudiation.  This  includes  providing  for  restoration  of  informa¬ 
tion  systems  by  incorporating  protection,  detection,  and  reaction  capabilities. 

Information  System:  a  discrete  set  of  information  resources  organized  for  the  collec¬ 
tion,  processing,  maintenance,  use,  sharing,  dissemination,  or  disposition  of  infor- 
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mation.  Since  infonnation  systems  can  comprise  multiple  products,  different  C&A 
schemes  for  products  and  systems  can  sometimes  conflict  with  one  another. 

Information  Technology  Product:  Package  of  IT  hardware,  software,  and  firmware  that 
provides  functionality  designed  for  use  or  incorporation  within  a  multiplicity  of  sys¬ 
tems.  An  IT  product  can  be  a  single  product  or  multiple  IT  products  configured  as 
an  IT  system,  network,  or  solution  to  meet  specific  customer  needs.  In  either  case, 
the  testing  occurs  in  a  testing  facility  or  a  client’s  site  under  laboratory  conditions, 
and  not  in  the  actual  operational  environment. 

Information  Technology  Security:  All  aspects  related  to  defining,  achieving,  and  main¬ 
taining  confidentiality,  integrity,  availability,  accountability,  authenticity,  and  reli¬ 
ability. 

Information  Technology  Security  Evaluation  Criteria:  a  compilation  of  the  informa¬ 
tion  that  must  be  provided  and  of  the  actions  that  must  be  taken  in  order  to  give 
grounds  for  the  confidence  that  evaluations  will  be  carried  out  effectively  and  to  a 
consistent  standard  throughout  an  evaluation  and  certification/validation  scheme. 

Information  Technology  Security  Evaluation  Facility:  an  accredited  EF,  licensed  or 
approved  to  perform  evaluations  within  the  context  of  a  particular  IT  security 
evaluation  and  certification/validation  scheme. 

Information  Technology  Security  Evaluation  Methods:  Compilation  of  the  methods 
that  need  to  be  used  by  Evaluation  Facilities  in  applying  ITSEC  in  order  to  give 
grounds  for  confidence  that  evaluations  will  be  carried  out  effectively  and  to  a  con¬ 
sistent  standard  throughout  an  evaluation  and  certification/validation  scheme. 

Information  Technology  Security  Policy:  Rules,  directives,  and  practices  that  govern 
how  assets,  including  sensitive  infonnation,  are  managed,  protected,  and  distributed 
within  an  organization  and  its  IT  systems. 

Input  task:  Tasks  related  to  the  management  of  all  required,  sponsor-supplied  evaluation 
evidence. 

Integrity:  Prevention  of  unauthorized  modification  of  information. 

Internal  communication  channel:  Communication  channel  among  different  parts  of  a 
TOE. 

Interpretation:  Expert  technical  judgment,  when  required,  regarding  the  meaning  or 
method  of  application  of  any  technical  aspect  of  the  criteria  or  the  methodology. 

Inter-TSF  transfers:  Communicating  data  between  the  TOE  and  the  security  functions 
of  other  trusted  IT  products. 

Iteration:  Use  of  an  element  more  than  once  with  varying  parameters. 

J 

Justification:  Analysis  leading  to  a  conclusion  but  which  is  more  rigorous  than  a  dem¬ 
onstration;  requires  significant  rigor  in  terms  of  very  carefully  and  thoroughly  ex¬ 
plaining  every  step  of  a  logical  argument. 
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M 

Monitoring  of  Evaluations:  Procedure  by  which  representatives  of  a  CB  observe  in 
progress  or  review  completed  evaluations  in  order  to  satisfy  themselves  that  an 
ITSEF  is  carrying  out  its  functions  in  a  proper  and  professional  manner. 

Monitoring  Phase:  Middle  of  an  assurance  maintenance  cycle  during  which  the  devel¬ 
oper  provides  evidence  at  one  or  more  points  that  assurance  of  the  TOE  is  being 
maintained  in  accordance  with  established  plans  and  procedures;  this  evidence  is 
independently  validated  by  an  evaluator. 

N 

National  Security  Information:  information  that  has  been  determined,  pursuant  to  Ex¬ 
ecutive  Order  12958  or  any  predecessor  order,  to  require  protection  against  unau¬ 
thorized  disclosure. 

National  Security  Systems:  any  telecommunications  or  information  system  operated  by 
the  United  States  Government,  the  function,  operation,  or  use  of  which — 

a.  involves  intelligence  activities; 

b.  involves  cryptologic  activities  related  to  national  security; 

c.  involves  command  and  control  of  military  forces; 

d.  involves  equipment  that  is  an  integral  part  of  a  weapon  or  weapons  system;  or 

e.  subject  to  subsection  (b  ),  is  critical  to  the  direct  fulfillment  of  military  or  in¬ 
telligence  missions. 

(see  section  C.2  for  context) 

National  Voluntary  Laboratory  Accreditation  Program:  the  U.S.  accreditation  author¬ 
ity  for  CCTLs  operating  within  the  NIAP  CCEVS. 

Network  Security:  Protection  of  information  systems  against  unauthorized  access  to  or 
modification  of  information,  whether  in  storage,  processing  or  transit,  and  against 
the  denial  of  service  to  authorized  users,  including  those  measures  necessary  to  de¬ 
tect,  document,  and  counter  such  threats. 

NIAP  Validation  Body:  A  governmental  organization  responsible  for  carrying  out  vali¬ 
dation  and  for  overseeing  the  day-to-day  operation  of  the  CCEVS. 


O 

Object:  Passive  entity  within  the  TOE  security  function  (TSF)  scope  of  control  (TSC) 
that  contains  or  receives  infonnation  and  upon  which  subjects  perform  operations. 


32  (b)  LIMITATION-  Subsection  (a)(5)  does  not  include  a  system  that  is  to  be  used  for  routine  administrative  and  business  applica¬ 
tions  (including  payroll,  finance,  logistics,  and  personnel  management  applications) 
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Observation  Decision:  A  response  to  an  Observation  Report  (OR).  The  observation  de¬ 
cision  (OD)  is  the  fonnal  documented  response  from  the  Validation  Body  that  pro¬ 
vides  clarification/guidance  to  the  CCTL  on  a  submitted  OR. 

Observation  Report:  A  report  issued  to  the  NIAP  Validation  Body  by  a  CCTL  or  spon¬ 
sor  identifying  specific  problems  or  issues  related  to  the  conduct  of  an  IT  security 
evaluation. 

Operations  Security:  the  implementation  of  standardized  operational  security  proce¬ 
dures  that  define  the  nature  and  frequency  of  the  interaction  between  users,  systems, 
and  system  resources,  the  purpose  of  which  is  to  (1)  maintain  a  system  in  a  known 
secure  state  at  all  times,  and  (2)  prevent  accidental  or  intentional  theft,  destruction, 
alteration,  or  sabotage  of  system  resources. 

Organizational  Security  Policy:  one  or  more  security  rules,  procedures,  practices,  or 
guidelines  imposed  by  an  organization  upon  its  operations. 

Output  Task:  Tasks  related  to  the  reporting  of  information  through  either  an  Observa¬ 
tion  Report  or  Evaluation  Technical  Report. 

P 

Package:  Set  of  either  functional  or  assurance  components  (e.g.,  an  EAL),  combined 
together  to  satisfy  a  subset  of  identified  security  objectives;  packages  are  intended 
to  be  used  to  build  PPs  and  STs. 

Preventive  Security  Objective:  Security  objectives  that  prevent  a  threat  from  being  car¬ 
ried  out  or  limit  the  ways  in  which  it  can  be  carried  out. 

Principal  Security  Assurance  Requirement:  Security  assurance  requirement  that  di¬ 
rectly  contributes  to  assuring  that  an  entity  meets  its  security  objectives. 

Principal  Security  Functional  Requirement:  Security  functional  requirement  that  di¬ 
rectly  satisfies  the  identified  security  objectives  of  the  TOE. 

Procedures:  step-by-step  "  how  to"  tasks  which  are  necessary  to  conduct  a  process  and 
meet  standards. 

Process:  Used  as  the  tag  for  stakeholder  class  of  individuals  who  help  develop  or  man¬ 
age  the  current  NIAP  process. 

Producer:  Used  as  the  tag  for  stakeholder  class  of  individuals  who  help  develop  IA  rele¬ 
vant  software. 

Product:  a  package  of  IT  software,  firmware  and/or  hardware,  providing  functionality 
designed  for  use  or  incorporation  within  a  multiplicity  of  systems. 

Profile:  Structure  that  characterizes  the  behavior  of  users  and  subjects;  it  represents  how 
users  and  subjects  interact  with  the  TSF. 

Profile  Metrics:  Ways  in  which  various  types  of  user  and  subject  activities  are  recorded 
and  measured  in  a  profile;  serves  as  input  to  pattern  recognition. 

Profile  Target  Group:  One  or  more  users  who  interact  with  the  TSF,  supposedly  accord¬ 
ing  to  historical  patterns  or  patterns  of  expected  behavior. 
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Protected  Information:  Information  gathered  or  obtained  during  an  evaluation,  the  un¬ 
authorized  disclosure  of  which  could  reasonably  be  expected  to  cause:  (1)  harm  to 
competitive  commercial  or  proprietary  interests,  (2)  a  clearly  unwarranted  invasion 
of  personal  privacy,  (3)  damage  to  national  security,  or  (4)  harm  to  an  interest  pro¬ 
tected  by  national  law,  legislation,  regulation,  policy,  or  official  obligation. 

Protection  Profile:  (1)  Formal  document  defined  in  the  CC  that  expresses  an  implemen¬ 
tation-independent  set  of  security  requirements  for  an  IT  product  that  meets  specific 
consumer  needs.  (2)  Complete  combinations  of  security  objectives  and  functional 
and  assurance  requirements  with  associated  rationale. 

Protocol:  a  set  of  semantic  and  syntactic  rules  that  detennine  the  behavior  of  entities  that 
interact. 

Prove:  Formal  analysis  in  the  mathematical  sense,  which  is  completely  rigorous  in  all 
ways. 

R 

Recognition  of  Common  Criteria  Certificates:  Acknowledgment  that  the  evaluation 
and  certification  processes  carried  out  by  compliant  CBs  appear  to  have  been  car¬ 
ried  out  in  a  duly  professional  manner  and  meet  all  the  conditions  of  the  CCRA  and 
the  intention  to  give  all  resulting  CC  Certificates  equal  weight. 

Reevaluation:  Evaluation  of  a  new  version  of  the  TOE  that  addresses  all  security¬ 
relevant  changes  made  to  the  certified  version  of  the  TOE  and  reuses  previous 
evaluation  results  where  they  are  still  valid. 

Reevaluation  Phase:  Completion  of  the  assurance  maintenance  cycle  in  which  an  up¬ 
dated  version  of  the  TOE  is  submitted  for  reevaluation  based  on  changes  affecting 
the  TOE  since  the  certified  version. 

Reference  Monitor:  Concept  of  an  abstract  machine  that  enforces  TOE  access  control 
policies. 

Reference  Validation  Mechanism:  Implementation  of  the  reference  monitor  concept 
that  possesses  the  following  properties:  tamper-proof,  always  invoked,  and  simple 
enough  to  be  subjected  to  thorough  analysis  and  testing. 

Refinement:  Addition  of  extra  details  to  an  element  when  it  is  used  in  a  PP  or  ST  (see 
also  Explicit  requirement  and  Extended). 

Reliability:  Property  of  consistent  intended  behavior  and  results. 

Request  for  Interpretation:  submitted  by  evaluation  authorities  to  CCIMB;  the  four 
types  are  (1)  perceived  error  such  that  some  content  in  the  CC  or  CEM  requires  cor¬ 
rection,  (2)  identified  need  for  some  additional  material  in  the  CC  or  CEM,  (3)  pro¬ 
posed  method  for  applying  the  CC  or  CEM  in  a  specific  circumstance  for  which  en¬ 
dorsement  is  sought,  and  (4)  request  for  information  to  assist  with  understanding  the 
CC  or  CEM. 

Residual  Risk:  (1)  Portion  of  risks  remaining  after  security  measures  has  been  applied. 
(2)  Risk  that  remains  after  safeguards  have  been  implemented. 
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Revocation:  Removal  of  the  accredited  status  of  a  laboratory  if  the  laboratory  is  found  to 
have  violated  the  terms  of  its  accreditation. 

Rigor:  Degree  of  structure  and  formality  applied  to  the  evaluation  by  the  evaluators. 

Risk:  (1)  Combination  of  the  likelihood  that  a  threat  will  be  carried  out  and  the  severity 
of  the  consequences  should  it  happens.  (2)  Potential  that  a  given  threat  will  exploit 
vulnerabilities  of  an  asset  or  group  of  assets  to  cause  loss  or  damage  to  the  assets. 

Risk  Assessment:  (1)  Process  of  analyzing  threats  to  and  vulnerabilities  of  an  IT  system 
and  the  potential  impact  the  loss  of  information  or  capabilities  of  a  system  would 
have;  the  resulting  analysis  is  used  as  the  basis  for  identifying  appropriate  and  cost- 
effective  countenneasures.  (2)  Process  of  identifying  security  risks,  determining 
their  magnitude,  and  identifying  areas  requiring  safeguards. 

Risk  Management:  (1)  Process  concerned  with  the  identification,  measurement,  con¬ 
trol,  and  minimization  of  security  risks  in  IT  systems  to  a  level  commensurate  with 
the  value  of  the  assets  protected.  (2)  The  entire  process  of  identifying,  controlling, 
and  eliminating  or  minimizing  uncertain  events  that  may  affect  IT  system  resources. 

Role:  Predefined  set  of  rules  establishing  the  allowed  interactions  between  a  user  and  the 
TOE. 


S 

Safeguard:  Practice,  procedure,  or  mechanism  that  reduces  risk. 

Scope  of  Accreditation:  Approved  test  methods  for  which  a  CCTL  has  been  accredited. 

Scope:  Portion  of  an  IT  product  or  system  that  is  being  evaluated. 

Security  Assurance:  Grounds  for  confidence  that  an  entity  meets  its  security  objectives. 

Security  Attribute:  Information  associated  with  users,  subjects,  and  objects  used  for  the 
enforcement  of  the  TSP. 

Security  Classification:  Labeling  applied  to  protected  infonnation  to  indicate  minimum 
standards  of  protection  that  need  to  be  applied  in  the  national  or  organizational  in¬ 
terest;  also  referred  to  as  protective  marking. 

Security  Flaw:  Condition  that  alone  or  in  concert  with  others  provides  an  exploitable 
vulnerability.  TSP  violations  that  occur  not  from  a  problem  with  the  hardware, 
software,  or  firmware  portion  of  a  TOE  but  from  a  problem  in  the  TOE  guidance  are 
also  recognized  as  security  flaws. 

Security  Objective:  Statement  of  intent  to  counter  identified  threats  and/or  satisfy  iden¬ 
tified  organization  policies  and  assumptions. 

Security  Target:  A  specification  of  the  security  required  (both  functionality  and  assur¬ 
ance)  in  a  Target  of  Evaluation  (TOE),  used  as  a  baseline  for  evaluation  under  the 
CC.  The  security  target  specifies  the  security  objectives,  the  threats  to  those  objec¬ 
tives,  and  any  specific  security  mechanisms  that  will  be  employed. 

Selection:  Specification  of  one  or  more  items  that  are  to  be  selected  from  a  list  given  in 
the  element  definition. 
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Semiformal:  Expressed  in  a  restricted  syntax  language  with  defined  semantics. 

Sensitive  Information:  Any  information  for  which  the  loss,  misuse,  or  unauthorized  ac¬ 
cess  to  or  modification  of  could  adversely  affect  the  national  interest  or  conduct  of 
federal  programs  or  the  privacy  to  which  individuals  are  entitled  under  the  Privacy 
Act  Section  552a  of  Title  5  USC,  but  which  has  not  been  specifically  authorized 
under  criteria  established  by  an  Executive  Order  or  an  Act  of  Congress  to  be  kept 
secret  in  the  interest  of  national  defense  or  foreign  policy. 

Security  Function:  Part  or  parts  of  the  TOE  that  have  to  be  relied  upon  for  enforcing  a 
closely  related  subset  of  the  rules  from  the  TSP 

Software  Assurance:  The  planned  and  systematic  set  of  activities  that  ensure  that 
software  life  cycle  processes  and  products  confonn  to  requirements,  standards,  and 
procedures. 

Sponsor:  The  person  or  organization  that  requests  a  security  evaluation  of  an  IT  product 
or  protection  profile. 

Standards:  documented  agreements  containing  technical  specifications  or  other  precise 
criteria  to  be  used  consistently  as  rules,  guidelines,  or  definitions  of  characteristics 
to  ensure  that  materials,  products,  processes,  and  services  are  fit  for  their  purpose. 

Strength  of  Function:  a  qualification  of  a  TOE  security  function  expressing  the  mini¬ 
mum  efforts  assumed  necessary  to  defeat  its  expected  security  behavior  by  directly 
attacking  its  underlying  security  mechanisms. 

Strength  of  Function-Basic:  Level  of  the  TOE  strength  of  function  where  analysis 
shows  that  the  function  provides  adequate  protection  against  causal  breach  of  the 
TOE  security  by  attackers  possessing  a  low  attack  potential. 

Strength  of  Function-High:  Level  of  the  TOE  strength  of  function  where  analysis 
shows  that  the  function  provides  adequate  protection  against  deliberately  planned  or 
organized  breach  of  the  TOE  security  by  attackers  possessing  a  high  attack  poten¬ 
tial. 

Strength  of  Function-Medium:  Level  of  the  TOE  strength  of  function  where  analysis 
shows  that  the  function  provides  adequate  protection  against  straightforward  or  in¬ 
tentional  breach  of  the  TOE  security  by  attackers  possessing  a  moderate  attack  po¬ 
tential. 

Subactivity:  Application  of  a  CC  assurance  component. 

Subject:  Active  entity  within  the  TSC  that  causes  operations  to  be  performed. 

Subtask:  Subdivision  of  a  task. 

Supporting  Security  Assurance  Requirement:  Security  assurance  requirement  that  in¬ 
directly  contributes  to  assuring  that  an  entity  meets  its  security  objectives. 

Supporting  Security  Functional  Requirement:  Security  functional  requirement  that 
does  not  directly  satisfy  security  objectives  for  the  TOE  but  which  provides  support 
to  the  principal  SFRs  and  hence  indirectly  helps  satisfy  TOE  security  objectives. 
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System  Integrity:  Property  that  a  system  performs  its  intended  function  in  an  unim¬ 
paired  manner,  free  from  deliberate  or  accidental  unauthorized  manipulation  of  the 
system. 

T 

Target  of  Evaluation:  An  IT  product,  part  of  an  IT  product  or  group  of  IT  products  and 
associated  documentation  that  is  the  subject  of  a  security  evaluation  under  the  CC. 

Target  of  Evaluation  Guidance:  Administrator  guidance,  user  guidance,  flaw  remedia¬ 
tion  guidance,  delivery  procedures,  and  installation,  generation,  and  start-up  proce¬ 
dures. 

Target  of  Evaluation  Security  Functions;  a  set  consisting  of  all  hardware,  software,  and 
firmware  of  the  TOE  that  must  be  relied  upon  for  the  correct  enforcement  of  the 
TSP. 

Target  of  Evaluation  Security  Functions  Scope  of  Control:  the  set  of  interactions  that 
can  occur  with  or  within  a  TOE  and  are  subject  to  the  rules  of  the  TSP. 

Target  of  Evaluation  Security  Functions  Interface:  the  set  of  interfaces,  whether  inter¬ 
active  man-machine  interfaces  or  application  program  interfaces,  through  which  re¬ 
sources  are  accessed  that  are  mediated  by  the  TSF  or  infonnation  obtained  from  the 
TSF. 

Target  of  Evaluation  Security  Policy:  a  set  of  rules  that  regulate  how  assets  are  man¬ 
aged,  protected,  and  distributed  within  a  TOE. 

Target  of  Evaluation  User:  Focal  point  in  the  user  organization  that  is  responsible  for 
receiving  and  implementing  fixes  to  security  flaws.  This  is  not  necessarily  an  indi¬ 
vidual  user  but  may  be  an  organizational  representative  who  is  responsible  for  the 
handling  of  security  flaws. 

Task:  Specifically  required  OEM  evaluation  work  that  is  not  derived  directly  from  a  CC 
requirement. 

Test  Method:  An  evaluation  assurance  package  from  the  CC,  the  associated  evaluation 
methodology  for  that  assurance  package  from  the  CEM,  and  any  technology- 
specific  derived  testing  requirements. 

Threat:  (1)  Any  circumstance  or  event  with  the  potential  to  harm  an  IT  system  through 
unauthorized  access,  destruction,  disclosure,  modification  of  data,  and/or  denial  of 
service.  (2)  Potential  danger  that  a  vulnerability  may  be  exploited  intentionally, 
triggered  accidentally,  or  otherwise  exercised.  (3)  A  potential  cause  of  an  unwanted 
incident,  which  may  result  in  harm  to  a  system  or  organization. 

Trusted  Channel:  Means  by  which  a  TSF  and  a  remote  trusted  IT  product  can  commu¬ 
nicate  with  necessary  confidence  to  support  the  TSP. 

Trusted  Path:  Means  by  which  a  user  and  a  TSF  can  communicate  with  necessary  confi¬ 
dence  to  support  the  TSP. 
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u 

Users:  ISO/IEC  recognizes  two  types  of  authorized  users:  (1)  local  or  remote  human  us¬ 
ers,  and  (2)  external  IT  entities.  Users  are  considered  to  be  outside  a  TOE  and  inter¬ 
act  with  a  TOE  through  the  TSFI. 

V 

Validation:  The  process  carried  out  by  the  NIAP  Validation  Body  leading  to  the  issue  of 
a  CC  certificate. 

Validated  Products  List:  A  publicly  available  document  issued  periodically  by  the 
NIAP  Validation  Body  giving  brief  particulars  of  every  IT  product  or  protection 
profile  which  holds  a  currently  valid  CC  certificate  awarded  by  that  body  and  every 
product  or  profile  validated  or  certified  under  the  authority  of  another  Party  for 
which  the  certificate  has  been  recognized. 

Validation  Report:  A  publicly  available  document  issued  by  the  National  Evaluation 
Authority  (in  the  U.S.  the  NIAP  Validation  Body  which  summarizes  the  results  of 
an  evaluation  and  confirms  the  overall  results,  (i.e.,  that  the  evaluation  has  been 
properly  carried  out,  that  the  CC,  the  Common  Evaluation  Methodology,  and 
scheme-specific  procedures  have  been  correctly  applied;  and  that  the  conclusions  of 
the  Evaluation  Technical  Report  are  consistent  with  the  evidence  adduced). 

Verification:  (1)  Confirmation  by  examination  and  provision  of  objective  evidence  that 
specified  requirements  have  been  fulfilled.  (2)  Process  of  comparing  two  levels  of 
an  IT  system  specification  for  proper  correspondence,  such  as  security  policy  model 
with  top-level  specification,  top-level  specification  with  source  code,  source  code 
with  object  code. 

Verify:  Independent  evaluator  actions;  similar  to  confirm  but  more  rigorous. 

Vulnerability:  Weakness  in  the  design,  operation,  or  operational  environment  of  an  IT 
system  or  product  that  can  be  exploited  to  violate  the  intended  behavior  of  the  sys¬ 
tem  relative  to  safety,  security,  and/or  integrity. 

W 

Work  Units:  Smallest  unit  of  an  evaluation  action;  derived  from  an  evaluator  action 
element  or  a  content  and  presentation  of  evidence  element. 

C.2  Some  context  for  terminology  used  in  this  report 

C.2.1  Cybersecurity  and  Related  Terms 

In  this  report,  the  terms  cybersecurity  and  information  assurance  are  used  interchangea¬ 
bly.  Cybersecurity  emerged  concurrently  with  the  DHS.  It  is  used  in  The  Strategy  to  Se¬ 
cure  Cyberspace  and  gained  fonnal  status  when  The  Department  of  Homeland  Security 
Authorization  Act  for  Fiscal  Year  2005  [DHS2005]  amended  the  Paperwork  Reduction 
Act  to  define  it  as: 
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the  prevention  of  damage  to,  the  protection  of,  and  the  restoration  of  computers, 
electronic  communications  systems,  electronic  communication  services,  wire 
communications,  and  electronic  communications,  including  information  con¬ 
tained  therein,  to  ensure  its  availability,  integrity,  authentication,  confidentiality, 
and  nonrepudiation. 

Whereas  cybersecurity  is  the  preferred  term  of  DHS,  information  assurance  is  the  pre¬ 
ferred  term  of  the  defense  and  intelligence  communities  where  it  has  been  well-defined 
for  a  long  time.  The  definitions  given  in  the  National  Information  Systems  Security 
(INFOSEC)  Glossary  [NST2000c],  Information  Assurance  (IA)  Awareness  Program, 
(AFI33-  204),  and  the  Industry  Advisory  Council,  Shared  Interest  Group  on  Information 
Assurance  are  all  similar: 

Conducting  those  operations  that  protect  and  defend  information  and  information 
systems  by  ensuring  availability,  integrity,  authentication,  confidentiality,  and 
non-repudiation.  This  includes  providing  for  restoration  of  information  systems 
by  incoiporating  protection,  detection,  and  reaction  capabilities. 

Both  cybersecurity  and  information  assurance  encompass  the  “five  pillars”  of  informa¬ 
tion  assurance — availability,  integrity,  authentication,  confidentiality,  and  nonrepudiation 
of  information  systems  as  well  as  the  concepts  of  protection  and  restoration.  Cybersecu¬ 
rity  refers  explicitly  to  computers  and  electronic  systems,  whereas  information  assurance 
refers  more  broadly  to  information  systems,  which  might  or  might  not  be  electronic. 

Information  security  is  often  erroneously  equated  with  information  assurance.  Unlike 
information  assurance,  information  security  is  not  well-defined  by  anyone 
[PETERSEN2004],  In  fact,  the  authoritative  source  of  information  systems  security  ter¬ 
minology,  the  National  Information  Systems  Security  (INFOSEC)  Glossary  [NST2000c] 
doesn’t  define  the  term  information  security,  preferring  the  terms  information  assurance, 
computer  security,  and  information  systems  security  (INFOSEC  or  ISS).  It  defines  com¬ 
puter  security  as: 

Measures  and  controls  that  ensure  confidentiality,  integrity,  and  availability  of  in¬ 
formation  system  assets  including  hardware,  software,  firmware,  and  information 
being  processed,  stored,  and  communicated. 

The  term  computer  security  dates  from  a  more  centralized,  single-system  approach  in 
contrast  to  the  networked,  distributed  systems  of  today.  With  the  growth  of  the  Internet 
especially,  the  term  network  security  has  become  more  widely  used.  [NST2000c]  equates 
network  security  to  INFOSEC/ISS  and  defines  them  as: 

Protection  of  information  systems  against  unauthorized  access  to  or  modification 
of  information,  whether  in  storage,  processing  or  transit,  and  against  the  denial  of 
service  to  authorized  users,  including  those  measures  necessary  to  detect,  docu¬ 
ment,  and  counter  such  threats. 

INFOSEC/ISS  does  not  encompass  the  five  pillars,  nor  does  it  refer  to  restoration  of  ser¬ 
vices.  For  these  reasons  and  to  minimize  confusion,  this  report  prefers  the  terms  cyberse¬ 
curity  and  information  assurance  to  all  these  other  terms. 
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C.2.2  National  Security  Systems 

Another  tenn  of  significance  to  this  report  is  national  security  systems  (NSS).  This  is  be¬ 
cause  the  security  requirements  of  NSSs  are  more  stringent  than  those  of  other  informa¬ 
tion  systems.  Section  5 142  of  the  Information  Technology  Management  Reform  Act  of 
1996  (Clinger-Cohen  Act)  [CCA  1996]  first  defined  national  security  systems  as: 

any  telecommunications  or  information  system  operated  by  the  United  States  Govern¬ 
ment,  the  function,  operation,  or  use  of  which — 

1)  involves  intelligence  activities; 

2)  involves  cryptologic  activities  related  to  national  security; 

3)  involves  command  and  control  of  military  forces; 

4)  involves  equipment  that  is  an  integral  part  of  a  weapon  or  weapons  system;  or 

5)  subject  to  subsection  (b33),  is  critical  to  the  direct  fulfillment  of  military  or  intel¬ 
ligence  missions. 

C.2.3  Critical  Infrastructure  Protection 


Several  recent  Government  Accountability  Office  (GAO)  reports  [GA02004c  and 
GA02004f]  have  underscored  the  vulnerability  of  the  critical  infrastructure  to  cyberse¬ 
curity  threats.  The  critical  infrastructure  is  defined  by  numerous  sources  ([ATIS2000], 
[EO13010],  [NST2000c],  [WH2003])  as: 

banking  and  finance,  energy,  chemical  sites,  transportation,  telecommunications, 
Government  facilities,  dams,  national  monuments  and  icons.  Cybersecurity  is  a 
key  element  of  infrastructure  protection. 

Although  most  critical  infrastructures  are  in  the  private  sector,  governments  at  all  levels 
perform  key  functions  that  depend  on  information  networks,  systems,  and  products. 
While  The  National  Strategy  for  the  Physical  Protection  of  Critical  Infrastructures  and 
Key  Assets,  [WH2003]  addressed  physical  security  of  the  critical  infrastructure,  The  Na¬ 
tional  Strategy  to  Secure  Cyberspace  [WH2003a]  addressed  the  protection  of  cyberspace. 
One  of  the  priorities  that  it  identifies  is  the  need  for  a  National  Cyberspace  Security 
Threat  and  Vulnerability  Program.  This  would  be  a  coordinated  effort  between  govern¬ 
ments  and  the  private  sector  to  identify  and  remediate  the  most  serious  cyber  vulnerabili¬ 
ties  through  collaborative  activities,  such  as  sharing  best  practices  and  evaluating  and  im¬ 
plementing  new  technologies.  The  NIAP  is  an  integral  part  of  this  and  another  strategy 
priority,  that  of  securing  governments’  cyberspace. 

The  terms  cyberspace,  cyber  security,  cyberwarfare,  and  cyberterrorism  do  not  appear  in 
[NST2000c],  having  entered  the  lexicon  since  September  11.  The  National  Infrastructure 


33  (b)  LIMITATION-  Subsection  (a)(5)  does  not  include  a  system  that  is  to  be  used  for  routine  administrative  and  business  applica¬ 
tions  (including  payroll,  finance,  logistics,  and  personnel  management  applications) 


C-18 


Protection  Center  (NIPC)  under  Director  Ron  Dick  [BERINAT02002]  defined  cyberter¬ 
rorism  as: 

a  criminal  act  perpetrated  through  computers  resulting  in  violence,  death 
and/or  destruction,  and  creating  terror  for  the  purpose  of  coercing  a  gov¬ 
ernment  to  change  its  policies. 

As  documented  by  the  GAO,  several  sources  point  to  an  escalation  in  the  cyberterrorism 
threat.  The  Federal  Bureau  of  Investigation  (FBI)  identifies  the  following  threats  to  the 
critical  infrastructure:  criminal  groups,  foreign  intelligence  services,  hackers,  hactivists, 
information  warfare,  insiders,  and  virus  writers.  Experts  agree  that  there  has  been  a 
steady  advance  in  the  level  of  sophistication  and  effectiveness  of  attack  technology. 

C.2.4  Quality-related  Terms 

From  1995  through  2003,  the  CERT1'  Coordination  Center  (CERT®/CC)  reported  12,946 
security  vulnerabilities  that  resulted  from  software  flaws.  This  is  significant  because  the 
potential  for  attack  increases  when  a  product  has  software  flaws.  Software  assurance 
methods  evaluate  products  for  flaws.  The  Institute  of  Electrical  and  Electronics  Engineers 
(IEEE)  Standard  Glossary  of  Software  Engineering  Tenninology  [IEEE2002]  defines 
software  assurance  as: 

The  planned  and  systematic  set  of  activities  that  ensure  that  software  life  cycle 
processes  and  products  conform  to  requirements,  standards,  and  procedures. 

As  an  example,  Title  III  of  the  E-Govemment  Act  (Public  Law  107-347)  entitled  the 
Federal  Information  Security  Management  Act  (FISMA),  requires  NIST  to  develop  risk- 
based  minimum  information  security  standards  for  systems  other  than  those  dealing  with 
national  security.  However,  there  is  often  confusion  regarding  the  tenns  standards,  guide¬ 
lines,  best  practices,  procedures,  and  protocols. 

The  International  Organization  for  Standardization  (ISO)  [ISO  1996a]  defines  standards 
as: 


documented  agreements  containing  technical  specifications  or  other  precise  crite¬ 
ria  to  be  used  consistently  as  rules,  guidelines,  or  definitions  of  characteristics  to 
ensure  that  materials,  products,  processes,  and  services  are  fit  for  their  purpose. 

As  a  recent  document  on  cybersecurity  Practices  and  Standards  Guidance  [CIDX2004] 
notes,  differentiating  between  a  standard  and  a  guideline  can  be  difficult.  It  says: 

Unfortunately,  no  litmus  test  can  be  applied  to  determine  when  a  guideline  may 
actually  be  a  standard.  The  name  assigned  to  the  document  or  meeting  may  be  of 
little  consequence.  It  is  the  degree  to  which  the  material  details  a  prescribed  set 
of  rules  for  procedures,  specifications,  materials,  design,  performance  or  opera¬ 
tion  (whether  voluntary  or  mandatory)  that  is  critical  in  determining  if  an  indus¬ 
try  standard  has  been  established. 

While  both  standards  and  guidelines  are  usually  voluntary,  standards  are  more 
susceptible  to  use  by  others  as  evidence  of  a  minimum  level  of  care,  despite  dis¬ 
claimers  to  the  contrary. 
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According  to  the  GAO  [GAO  1997],  best  practices  refer  to  the 

processes,  practices,  and  systems  identified  in  public  and  private  organizations 
that  performed  exceptionally  well  and  are  widely  recognized  as  improving  an  or¬ 
ganization's  performance  and  efficiency  in  specific  areas.  Successfully  identify¬ 
ing  and  applying  best  practices  can  reduce  business  expenses  and  improve  organ¬ 
izational  efficiency. 

The  Sacramento  County  Office  of  Quality  and  Strategic  Planning  [QUALITYn.d.]  de¬ 
fines  procedures  as: 

step-by-step  "  how  to"  tasks  which  are  necessary  to  conduct  a  process  and  meet 
standards. 

The  IEEE  Portable  Applications  Standards  Committee  [IEEE  1995]  defines  protocol  as: 

a  set  of  semantic  and  syntactic  rules  that  determine  the  behavior  of  entities  that 
interact. 

C.2.5  Certification  and  Accreditation 


Many  Federal  agencies  are  using  certification  and  accreditation  (C&A)  is  to  assure  that 
products  and  systems  are  secure.  In  addition  to  requiring  NIST  to  develop  security  stan¬ 
dards,  FISMA  also  requires  C&A  of  information  systems. 

ISO  defines  certification  as: 

the  procedure  by  which  a  third  party  gives  written  assurance  that  a  product,  proc¬ 
ess,  or  service  conforms  to  specified  requirements  or  standards. 

ISO  defines  accreditation  as: 

the  procedure  by  which  an  authoritative  body  gives  formal  recognition  that  a  body 
or  person  is  competent  to  carry  out  specific  tasks.  In  the  context  of  certification, 
an  accreditation  body  might  accredit  a  certification  body,  such  as  a  testing  labora¬ 
tory,  as  competent  to  carry  out  certification  activities — in  a  sense,  certifying  the 
certifiers. 

C&A  can  either  be  product-  or  information  system- specific.  The  CC  defines  a  product  as: 

a  package  of  IT  software,  firmware  and/or  hardware,  providing  functionality  de¬ 
signed  for  use  or  incorporation  within  a  multiplicity  of  systems. 

Various  sources  [Title  44  U.S.C.,  Section  3502  and  OMB 1996]  define  an  information 
system  as: 

a  discrete  set  of  information  resources  organized  for  the  collection,  processing,  mainte¬ 
nance,  use,  sharing,  dissemination,  or  disposition  of  information. 

Since  infonnation  systems  can  comprise  multiple  products,  different  C&A  schemes  for 
products  and  systems  can  sometimes  conflict  with  one  another. 
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C.2.5.1  Product  C&A 


Although  C&A  is  applied  to  product  evaluations,  for  the  purposes  of  this  report,  product 
or  algorithm  evaluation  will  be  the  preferred  term  when  referring  to  evaluations  below 
the  system  level.  The  NSTISSP  11  [NST2002;  NST2003]  policy  directs  departments  and 
agencies  of  the  U.S.  Federal  government  to  acquire  only  COTS  IA  and  IA-enabled  IT 
products  (to  be  used  on  systems  entering,  processing,  storing,  displaying,  or  transmitting 
national  security  information)  which  have  been  evaluated  and  validated  in  accordance 
with  criteria,  schemes,  or  programs  of  the: 

1 .  ISO/IEC 1 5408,  Common  Criteria, 

2.  The  NIAP  evaluation  and  validation  program,  and 

3.  FIPS  validation  program.  [NST2003]. 

To  avoid  having  two  different  standards,  one  for  national  security  systems  and  one  for  the 
rest  of  the  systems  in  DoD,  DoD  Directive  (DoDD)  8500.1  [DoD2002a]  requires  «//DoD 
systems  to  meet  NSTISSP  1 1  requirements. 

The  rest  of  the  Federal  government  is  not  required  to  evaluate  and  validate  the  products 
they  use. 

C.2.5.2  Information  System  C&A 

Although  C&A  is  applied  to  product  evaluations,  for  the  purposes  of  this  report,  C&A 
will  be  the  preferred  tenn  when  referring  to  evaluations  at  the  system  level.  To  address 
information  system- level  security  concerns,  four  C&A  processes  have  emerged: 

1.  DoD  Infonnation  Technology  Security  Certification  and  Accreditation  Process 
(DITSCAP)  [DoD  1997]  which  applies  to  all  DoD  entities, 

2.  National  Infonnation  Assurance  Certification  and  Accreditation  Process 
(NIACAP)  which  applies  to  National  Security  Systems, 

3.  Protecting  Sensitive  Compartmented  Information  Within  Information  Systems, 
Director  of  Central  Intelligence  Directive  6/3  (DCID  6/3)  which  applies  to  the  In¬ 
telligence  Community,  and 

4.  Guide  for  the  Security  Certification  and  Accreditation  of  Federal  Infonnation  Sys¬ 
tems,  NIST  Special  Publication  SP  800-37,  which  applies  to  all  U.S.  government 
Executive  Branch  departments,  agencies,  and  their  contractors  and  consultants. 

The  first  three  are  mandatory  for  their  communities,  while  the  NIST  SP  provides  guide¬ 
lines  for  certifying  and  accrediting  information  systems  supporting  the  executive  agencies 
of  the  Federal  government. 

DITSCAP,  upon  which  NIACAP  is  based,  was  developed  first.  It  is  a  four-stage  process: 

•  Phase  I  -  Definition  -  Defining  and  documenting  mission,  function,  re¬ 
quirements,  and  capabilities  culminating  in  a  draft  system  security  au¬ 
thorization  agreement  (SSAA) 
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•  Phase  II  -  Verification  -  Verifying  that  the  evolving  or  modified  system’s 
compliance  with  the  SSAA 

•  Phase  III  -  Validation  -  Validating  the  SSAA  using  vulnerability  and 
penetration  testing,  resulting  in  full,  interim,  or  withheld  accreditation 

•  Phase  IV  -  Post  accreditation  -  Monitoring  and  maintenance  to  ensure 
continued  security. 

The  goal  of  these  processes  is  to  introduce  integrated  security  into  the  life  cycle  of  IT  sys¬ 
tems  to  minimize  risks  in  shared  infrastructures.  Both  DITSCAP  and  NIACAP  define 
certification  as: 

The  comprehensive  evaluation  of  the  technical  and  non  technical  security  features 
of  an  information  system  and  other  safeguards,  made  in  support  of  the  accredita¬ 
tion  process  to  establish  the  extent  to  which  a  particular  design  and  implementa¬ 
tion  meets  a  set  of  specified  security  requirements. 

Their  shared  definition  of  accreditation  is: 

Formal  declaration  by  the  Designated  Accrediting  Authority  (DAA)  that  an  IT 
system  is  approved  to  operate  in  a  particular  security  mode  using  a  prescribed  set 
of  safeguards  at  an  acceptable  level  of  risk. 

As  part  of  its  FISMA  Implementation  Process,  NIST  developed  SP  800-37  [NIST2002b]. 
Its  definitions  of  C&A  are  almost  identical  to  those  of  DITSCAP  and  NIACAP: 

Certification:  A  comprehensive  assessment  of  the  management,  operational,  and 
technical  security  controls  in  an  information  system,  made  in  support  of  security 
accreditation,  to  detennine  the  extent  to  which  the  controls  are  implemented  cor¬ 
rectly,  operating  as  intended,  and  producing  the  desired  outcome  with  respect  to 
meeting  the  security  requirements  for  the  system. 

Accreditation:  The  official  management  decision  given  by  a  senior  agency  official 
to  authorize  operation  of  an  infonnation  system  and  to  explicitly  accept  the  risk  to 
agency  operations  (including  mission,  functions,  image,  or  reputation),  agency  as¬ 
sets,  or  individuals,  based  on  the  implementation  of  an  agreed-upon  set  of  security 
controls. 

In  keeping  with  FISMA  requirements,  the  NIST,  DITSCAP,  and  NIACAP  definitions  of 
accreditation  emphasize  the  concept  of  risk,  which  the  ISO  definition  does  not. 
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Annex  D  Policy 

This  appendix  contains  a  detailed  discussion  of  the  policies  and  other  documents  relating 
to  the  NIAP.  The  policies  are  grouped  in  roughly  hierarchical  order  based  on  the  five 
themes  of  cybersecurity,  Standards/Guidelines,  Education  &  Training,  Research,  and  Ac¬ 
quisition.  Certain  documents  may  appear  multiple  times  to  account  for  multiple  themes 
within  these  documents.  The  detailed  description  of  each  theme  is  contained  within 
Chapter  3,  but  a  short  summary  will  precede  the  documents  so  that  the  reader  need  not  go 
back  and  forth  between  the  chapter  and  this  appendix.  The  Tab  A  at  this  end  of  this  An¬ 
nex  provides  a  broad  overview  of  the  policies  and  the  thematic  areas  covered,  and  illus¬ 
trate  the  complexity  of  the  policy  landscape. 

D.l  Cybersecurity 

The  first  theme  is  cybersecurity.  This  word  is  frequently  used  interchangeably  with  in¬ 
formation  security  and  information  assurance.  It  is  a  component  of  the  critical  infrastruc¬ 
ture.  This  theme  will,  therefore,  address  the  policies  surrounding  cybersecurity  as  part  of 
the  critical  infrastructure.  The  grouping  that  follows  reflects  the  explicit  relationships 
among  the  policies. 

D.1.1  Computer  Security  Act  (CSA)  198734 

The  CSA  was  Congress’s  first  attempt  to  specify  actions  the  Federal  government  needed 
to  take  to  address  a  real  and  growing  problem.  It  states  the  goal  of  Congress  is  to  im¬ 
prove  the  security  and  privacy  of  sensitive  information  in  Federal  computer  systems. 
Congress  directed  the  following  actions:  (1)  assigned  the  NIST  the  responsibility  for  de¬ 
veloping  standards  and  guidelines  (spelled  out  in  detailed  later  on  in  this  section)  with 
advice  from  the  NSA  where  appropriate;  (2)  provided  a  mechanism  for  promulgation  of 
these  standards  and  guidelines,  including  specifying  which  are  mandatory  and  which  are 
voluntary;  (3)  required  establishment  of  security  plans  for  Federal  computer  systems  that 
contain  sensitive  information;  and  (4)  required  mandatory  periodic  training  for  all  per¬ 
sonnel  involved  in  management,  use,  or  operation  of  Federal  computer  systems  that  con- 
tain  sensitive  information.  '  The  documents  that  follow  in  this  section  were  issued  pri¬ 
marily  in  response  to  the  requirements  laid  out  by  this  statute. 

D.1.2  National  Security  Directive  4236  1990  (NSD  42), 

NSD  42  established  policy  and  procedures  intended  to  ensure  that  infonnation  housed  in 
National  Security  Systems  (NSS)  cannot  be  exploited  for  hostile  purposes.  The  specific 
policy  states  that: 

a.  U.S.  Government  (USG)  national  security  systems  shall  be  secured  by  such 
means  as  are  necessary  to  prevent  compromises,  denials  or  exploitation; 


34  Public  Law  100-235,  40  USC  759,  “Computer  Security  Act  of  1987”. 

35  Ibid,  Section  2. 

36  National  Security  Directive  42,  “National  Policy  for  the  Security  of  National  Security  Telecommunica¬ 
tions  and  Information  Systems”,  5  July  1990. 
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b.  Federal  agencies  shall  require  that  national  security  systems  operated  and 
maintained  by  U.S.  Government  contractors  likewise  be  secured37 

This  document  defined  NSS  for  the  first  time  as  those  telecommunications  and  informa¬ 
tion  systems  operated  by  the  USG,  its  contractors  or  agents,  that  contain  classified  infor¬ 
mation  or  involve  intelligence  activities  related  to  national  security,  command  and  control 
of  military  forces,  equipment  integral  part  of  a  weapon  or  weapon  system,  or  equipment 
critical  to  the  direct  fulfillment  of  military  or  intelligence  missions.  It  created  an  organ¬ 
izational  structure  to  guide  efforts  to  secure  NSS  from  exploitation,  establishes  a  mecha¬ 
nism  to  develop  and  disseminate  policy,  and  assigns  roles  and  responsibilities  for  imple¬ 
mentation.  The  National  Security  Council/Policy  Coordinating  Committee  (NSC/PCC) 
for  the  National  Security  Telecommunications  and  Information  Systems  was  assigned 
responsibility  for  overseeing  implementation  of  NSD-42  and  for  developing  policy  rec¬ 
ommendations  and  providing  guidance  to  the  National  Security  Telecommunications  and 
Information  Systems  Security  Committee  (NSTISSC).  NSTISSC,  later  renamed  the 
Committee  for  National  Security  Systems  (CNSS)  by  Executive  Order  13231,  in  turn, 
was  charged  with  developing  operating  policies,  procedures,  guidelines,  instructions  and 
standards  to  implement  NSD-42.  One  of  this  Directive’s  stated  objectives  is  the  creation 
of  “a  technical  base  within  the  U.S.  Government  to  achieve  this  security,  and  initiatives 
with  the  private  sector  to  maintain,  complement,  or  enhance  that  government  technical 
base  and  to  ensure  information  systems  security  products  are  available  to  secure  NSS.”38. 

D.1.3  NSTISSC/CNSS  Issuances 

The  Secretary  of  Defense  (SECDEF)  is  designated  as  the  Executive  Agent  for  National 
Security  Systems.  The  Director,  NSA  (DIRNSA)  is  designated  as  the  National  Manager 
for  National  Security  Systems  and  is  responsible  for  the  permanent  secretariat  for  the 
Committee  for  National  Security  Systems  (formerly  the  NSTISSC).39  Through  the  CNSS 
Issuances  (instructions,  directives,  manuals  and  advisory  memoranda),  SECDEF  and  the 
National  Manager  provide  policy  and  guidance  to  on  cybersecurity  to  Federal  agencies 
who  are  members  of  the  NSS.  A  complete  list  of  these  issuances  can  be  found  at  the  fol¬ 
lowing  web  site:  http://www.nstissc.gov/html/library.html. 

D.1.4  Executive  Order  1233340  of  1981  (EO  12333) 

This  document  provided  policy  and  guidance  to  the  intelligence  community  (IC)  and 
spelled  out  responsibilities  of  the  various  entities  within  this  community  as  well  as  identi¬ 
fying  what  Federal  components  are  considered  to  be  part  of  this  community.  Addition¬ 
ally,  it  designated  the  Director  of  Central  Intelligence  (DCI)  as  the  head  of  this  commu¬ 
nity  and  gave  that  position  the  authority  to  provide  policy  and  guidance  to  the  IC,  includ¬ 
ing  common  security  and  access  standards  for  managing  and  handling  foreign  intelli¬ 
gence  systems,  information  and  products. 


37  Ibid,  para  2. 

38NSD  42,  paral.b. 

39  NSD  42 

40  EO  12333,  “United  States  Intelligence  Activities”,  4  December  1981. 
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D.1.5  Executive  Order  1295841  of  1995  (EO  12958) 

This  order  prescribed  a  “uniform  system  for  classifying,  safeguarding  and  declassifying 
national  security  information.”  It  provided  definitions  for  national  security  and  classified 
national  security  and  instructions  on  classification/declassification  standards  and  authori¬ 
ties.  It  also  directed  agency  heads  to  establish  uniform  procedures  to  ensure  that  Infor¬ 
mation  Technology  (IT)  systems,  including  networks  and  telecommunications  systems 
that  collect,  create,  communicate,  compute,  disseminate,  process  or  store  classified  in¬ 
formation  have  controls  that  (1)  prevent  access  by  unauthorized  persons,  and  (2)  ensure 
the  integrity  of  the  infonnation. 

D.1.6  Executive  Order  13010  Critical  Infrastructure  Protection42  of  1996  (EO 
13010)/Presidential  Decision  Directive  63  1998  (PDD  63) 

These  two  documents  were  the  seminal  documents  in  the  Administration’s  efforts  to  de¬ 
fine  and  address  the  issue  of  Critical  Infrastructure  Protection.  EO  13010  was  the  origi¬ 
nal  executive  order  chartering  the  President’s  Commission  on  Critical  Infrastructure  Pro¬ 
tection  to  assess  “the  scope  and  nature  of  the  vulnerabilities  of,  and  threats  to,  critical  in¬ 
frastructures”43  and  make  recommendations  on  what  the  Administration  should  do  about 
the  issue.  The  EO  also  established  the  Infrastructure  Protection  Task  Force  within  the 
Department  of  Justice  to  handle  coordination  of  existing  infrastructure  protection  efforts 
until  the  Commission  made  its  recommendations.  The  Commission  issued  its  report  in 
1997,  where  it  identified  a  number  of  sectors,  including  the  information  and  communica¬ 
tions  sector,  and  described  vulnerabilities,  findings  and  recommendations44. 

The  Clinton  Administration’s  response  to  the  recommendations  of  the  Commission  was 
the  issuance  of  Presidential  Decision  Directive  6345  (PDD  63).  This  document  defined, 
for  the  first  time,  the  critical  infrastructure,  lead  agencies  and  expectations  of  the  private 
sector  and  government  to  begin  to  address  the  risks  posed  to  U.S.  critical  infrastructures. 
The  Department  of  Commerce  was  designated  the  lead  agency  for  the  Information  and 
Communications  Sector.  For  the  Federal  government,  it  stated  that  department  and  agen¬ 
cies  were  responsible  for  protecting  their  own  critical  infrastructures,  especially  their  cy¬ 
ber-based  systems.  It  called  for  vulnerability  assessments  on  government  computer  and 
physical  systems  and  for  departments  and  agencies  to  develop  plans  for  protecting  their 
critical  infrastructures,  including  their  cyber-based  systems.  The  plans  were  to  be  imple¬ 
mented  no  later  that  two  years  from  the  date  of  the  document  and  updated  every  two 
years.  This  document  also  set  up  a  number  of  new  entities:  the  National  Infrastructure 
Protection  Center  (NIPC)  at  the  FBI;  Infonnation  Sharing  and  Analysis  Centers  (ISAC) 
set  up  by  sector  coordinators  with  the  private  sector;  and  a  number  of  follow-on  studies. 


EO  12958,  “Classified  National  Security  Information”,  17  April  1995. 

42  EO  13010,  “Critical  Infrastructure  Protection”,  15  July  1996. 

43  Ibid,  Sec  4  Mission. 

44  PCCIP,  Critical  Foundations:  Protecting  America’s  Infrastructures,  October  1997,  Appendix  A  “Sector 
Summary  Reports,  pp  A-2  through  A- 10. 

45  PDD63,  “Critical  Infrastructure  Protection”,  22  May  1998. 
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It  also  created  the  position  of  a  National  Coordinator  who  reported  to  the  President 
through  the  Assistant  to  the  President  for  National  Security  Affairs.46  The  significance  of 
this  activity  to  the  NIAP  was  that,  for  the  first  time,  cybersecurity  was  identified  as  an 
issue  of  concern  to  the  Federal  government  and  the  process  to  evaluate  software  embod¬ 
ied  in  the  NIAP  process  is  an  essential  element  contributing  to  cybersecurity. 

D.1.7  Executive  Order  1323147  of  2001(EO  13231) 

This  document  sets  policy  for  protecting  information  systems  for  critical  infrastructures. 
It  states  that: 

It  is  the  policy  of  the  United  States  to  protect  against  disruption  of  the  operation 
of  information  systems  for  critical  infrastructure  and  thereby  help  to  protect  the 
people,  economy,  essential  human  and  government  services,  and  national  security 
of  the  United  States,  and  to  ensure  that  any  disruptions  that  occur  are  infrequent, 
of  minimal  duration,  and  manageable,  and  cause  the  least  damage  possible.  The 
implementation  of  this  policy  shall  include  a  voluntary  public-private  partner¬ 
ship,  involving  corporate  and  nongovernmental  organizations. 

This  EO  also  renamed  NSTISSC  to  the  CNSS,  but  left  its  chairmanship  and  charter  in¬ 
tact.  It  established  a  Critical  Infrastructure  Protection  Board  (CIP  Board)  chaired  by  the 
President’s  Special  Advisor  for  Cyberspace  Security.  This  board  chartered  a  number  of 
standing  committees,  as  well  as  incorporating  two  existing  committees,  one  of  which  was 
the  CNSS.48  The  public-private  partnership  called  out  in  this  EO  is  a  partnership  be¬ 
tween  the  government  and  private  sector  owners/operators  of  the  critical 
infrastructures. 49 . 

D.1.8  Paperwork  Reduction  Act  of  1980  as  amended  by  the  Paperwork  Reduc¬ 
tion  Act  of  1995  (PRA)50 

The  purpose  of  this  statute  was  a  nearly  complete  revision  of  the  original  PRA  of  1980  to 
provide  comprehensive  direction  to  the  Federal  government  to  become  more  responsive 
and  accountable  for  reducing  the  burden  of  Federal  paperwork  on  the  public.  It  also  ad¬ 
dressed  a  number  of  other  information  management  policies,  including  ensuring  that  the 
creation,  collection,  maintenance,  use,  dissemination,  and  disposition  of  information  for 
or  by  the  Federal  government  is  consistent  with  applicable  laws  relating  to  the  security  of 
information,  including  CSA  1987.  The  Act  created  an  Office  of  Information  and  Regula¬ 
tory  Affairs  in  OMB  whose  head  was  responsible  for  accomplishing  those  oversight  ac- 


46  Although  this  position  still  exists,  the  majorities  of  the  functions  were  assigned  to  the  Department  of 
Homeland  Security  by  the  HSA  2002  and  are  performed  in  the  National  Cyber  Security  Division  (NCSD) 
of  the  Directorate  for  Information  Analysis  and  Infrastructure  Protection. 

47  EO  13231,  “Critical  Infrastructure  in  the  Information  Age,”  18  Oct  2001. 

48  ibid,  Sections  2  &  7. 

49  This  board,  along  with  the  standing  committees  it  established,  was  disestablished  by  a  later  EO.  The  two 
preexisting  committees,  CNSS  and  NCS’s  Committee  of  Principles  (COP),  remained  in  existence  under 
their  original  charters.  EO  13284,  “Amendment  of  Executive  Orders,  and  other  Actions,  in  Connection 
With  the  Establishment  of  the  Department  of  Homeland  Security”,  23  January  2003,  Section  2. 

50  Public  Law  104-13,  “Paperwork  Reduction  Act  of  1995”,  22  May  1995.  (44USC  Chapter  35) 
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tivities  assigned  to  the  Director,  OMB,  for  infonnation  resources.  These  activities  in¬ 
clude  development,  coordination  and  oversight  of  the  implementation  of  Federal  infonna¬ 
tion  resources  management  policies,  principles,  standards  and  guidelines,  for,  among 
other  things,  the  security  of  information.  Federal  agencies  were  responsible  for  carrying 
out  the  agency’s  information  resources  management  activities  to  improve  agency  produc¬ 
tivity,  efficiency  and  effectiveness  and  agency  heads  were  directed  to  designate  a  senior 
official  to  carry  out  these  responsibilities.  This  position  in  later  statutes  became  the  CIO. 
Agencies  were  also  directed  to  implement  and  enforce  policies,  procedures,  standards, 
and  guidelines  on  security  for  the  agencies  and  to  identify  and  to  apply  appropriate  secu¬ 
rity  measures  as  indicated  by  their  risk  management  assessment  consistent  with  CSA 
1987. 

D.1.9  Clinger-Cohen  Act  1996(CCA)  51 

This  statute  (originally  titled  “Information  Technology  Management  Reform  Act”)  was  a 
significant  effort  on  the  part  of  Congress  to  make  substantial  changes  in  how  the  Federal 
government  acquired  and  managed  information  technology.  The  CCA  gave  the  Office 
of  Management  and  Budget  (OMB)  authority  over  the  Federal  agencies  IT  programs  and 
required:  the  establishment  of  a  capital  planning  process;  use  of  performance  and  results- 
based  management;  designation  of  federal  agency  Chief  Information  Officers  (CIOs);  and 
a  number  of  annual  reports  on  progress  in  implementing  the  statute. 

The  CCA  directed  agency  CIOs  to  establish  an  IT  capital  planning  process,  of  which  in¬ 
formation  security  is  a  component,  and  modified  the  requirement  to  develop  and  imple¬ 
ment  information  security  plans,  originally  required  by  CSA  1987,  by  providing  more  de¬ 
tails  on  what  should  be  included  in  the  plan.  Additionally,  it  reiterated  the  direction  to  the 
Secretary  of  Commerce  to  issue  standards  and  guidance  on  information  security  (devel¬ 
oped  by  NIST)  originally  required  by  CSA  1987,  and  included  the  determination  of 
which  standards  should  be  made  mandatory  and  which  should  be  made  voluntary.  It  gave 
the  Federal  agency  heads  the  authority  to  employ  more  stringent  standards  and  a  waiver 
process  for  those  standards  detennined  by  the  Secretary  of  Commerce  to  be  mandatory 
standards.  The  statute  also  defined  national  security  systems  in  law  for  the  first  time  and 
specified  what  part  of  the  act  applied  to  these  systems.  Finally,  the  CCA  directed  OMB 
to  evaluate  Federal  agency  programs  in  infonnation  technology  management  and,  in  par¬ 
ticular,  to  ensure  that  agency  infonnation  security  policies,  procedures  and  practices  were 
adequate. 


51  Public  Law  104-106,  Division  E,  “Information  Technology  Management  Control  Act  of  1996”,  10  Feb 
1996,  sections  5113,  5131,  5141,  5142,  and  5607. 

52  H.R.  Report  104-450  to  accompany  S.  1 124,  Division  E-  “Information  Technology  Management  Re¬ 
form”,  pages  972-982  discusses  Congress’s  frustration  with  the  lack  of  progress  within  the  Federal  gov¬ 
ernment  in  the  area  of  information  technology  and  their  intent  in  giving  the  Director,  OMB,  significant 
oversight  authority  in  establishing  performance -based  and  results-based  management  for  IT. 
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D.1.10  Executive  Order  13011  (EO  13011)53 

This  EO  was  issued  to  provide  policy  direction  to  Federal  agencies  to  assist  their  imple¬ 
mentation  of  the  requirements  to  improve  the  acquisition  and  management  of  information 
technology  as  required  by  CCA  and  the  Paperwork  Reduction  Act  of  1995  (PRA).  The 
EO  laid  out  the  administration’s  policy  regarding  the  management  of  Federal  IT  systems; 
defined  the  responsibilities  of  agency  heads;  chartered  the  Chief  Information  Officer’s 
Council,  the  Government  Information  Technology  Services  Board,  and  the  Information 
Technology  Resources  Board  (including  membership  &  responsibilities),  and  specific 
responsibilities  of  designated  offices  for  this  policy.  It  reiterated  agency  head  responsi¬ 
bilities  for  ensuring  that  their  information  security  policies,  procedures  and  practices  were 
adequate  and  provided  additional  clarification  of  what  constituted  a  national  security  sys¬ 
tem.  This  EO  also  directed  OMB  to  provide  to  the  Federal  agencies,  implementation 
guidance  for  this  EO  and  on  the  management  of  information  resources. 

D.1.11  Office  of  Management  &  Budget  Circular  A-130  (OMB  Cir  A-130),  Re¬ 
vised,  Transmittal  Memorandum  No.  4,  28  November  2000.  54 

As  directed  by  EO  13011,  OMB  issued  this  Circular  to  provide  implementing  policy  to 
Federal  agencies  for  several  statutes,  including  the  PRA,  CCA  and  the  CSA.  To  ensure 
security  in  information  systems  as  required  by  the  three  statutes  cited,  OMB  directed 
agencies  to  incorporate  security  into  the  architecture  of  their  information  and  systems, 
and  fund  and  manage  security  through  plans  built  into  the  life-cycle  budgets  for  informa¬ 
tion  systems.  This  document  also  laid  out  responsibilities  for  all  Federal  agencies,  as 
well  as  specific  responsibilities  for  certain  government  agencies  (such  as  DoD,  Govern¬ 
ment  Services  Administration  (GSA),  and  the  National  Archives  and  Records  Administra¬ 
tion  (NARA)),  as  well  as  OMB’s  role  in  ensuring  infonnation  security  for  Federal  sys¬ 
tems.  To  accomplish  these  objectives,  OMB  provided  detailed  guidance  in  Appendix  III 
of  this  circular.55  As  described,  agencies  must  ensure  that  information  was  protected 
commensurate  with  the  risk  and  magnitude  of  the  hann  that  would  result  from  the  loss, 
misuse,  or  unauthorized  access  to  or  modification  of  such  information.  Within  their  IT 
capital  planning  process,  agencies  were  required  to  establish  oversight  mechanisms  to 
evaluate  and  ensure  the  continued  security  of  their  systems  and  data.  As  part  of  this, 
agencies  must  develop  a  security  plan,  based  on  criteria  specified  in  this  appendix. 

D.l  12  DoD  Issuances 

The  authorities  of  the  Director,  OMB,  described  above,  are  delegated  to  SECDEF  in  the 
case  of  systems  operated  by  DoD,  contractors  for  DoD  or  another  entity  on  behalf  of 
DoD  that  processes  any  infonnation  the  unauthorized  access,  use,  disclosure,  disruption, 
modification,  or  destruction  of  which  would  have  a  debilitating  impact  on  the  mission  of 


53  Executive  Order  13011,  “Federal  Information  Technology”,  16  July  1996. 

54  OMB  Circular  A-130,  “Management  of  Federal  Information  Resources”,  Revised,  transmittal  No.  4,  20 
November  2000. 

55  Appendix  III,  OMB  Circular  A- 130, ’’Security  of  Federal  Automated  Information  Resources”,  November 
2000. 
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DoD.56  DoD  has  been  very  active  in  providing  policy  to  its  subordinate  organizations. 
The  primary  document  for  cybersecurity  (information  assurance  in  DoD’s  tenninology)  is 
DoD  Directive  8500.1. 57  This  document  describes  the  overall  DoD  policy  for  this  area 
and  assigns  responsibility  for  execution  of  the  program  to  the  DoD  CIO.  The  companion 

58 

instruction,  DoD  Instruction  8500.2  provides  details  for  the  execution  of  the  programs. 

A  lower  level  document,  Chairman  of  the  Joint  Chiefs  of  Staff  Instruction  6510.01D  de¬ 
scribes  the  IA  program  for  the  Joint  Staff,  combatant  commands  and  military  services.59 
There  are  other  documents  that  contain  policy  for  the  Department  in  specific  areas,  but 
these  three  are  the  major  ones  outlining  the  entire  IA  program. 

D.1.13  Intelligence  Community 

As  this  document  was  being  written,  the  leadership  of  the  Intelligence  Community  (IC) 
was  undergoing  significant  change  in  light  of  the  post-911  concerns  about  intelligence 
reform.  It  is  outside  the  scope  of  this  study  to  present  an  in-depth  assessment  of  the  on¬ 
going  changes  that  are  occurring.  For  the  purposes  of  this  study,  discussion  is  limited  to 
four  documents  that  establish  the  responsibilities  for  oversight  of  the  community  IT  and 
cybersecurity.  The  four  documents  include: 

D.  1.13.1  National  Security  Act  of  1947  (NSA  1947) 

D. 1,13.2  E012333,  United  States  Intelligence  Activities  1981 

D. 1.13.3  EO  13355,  Strengthened  Management  of  the  Intelligence  Community, 
2004 

D. 1.13.4  Intelligence  Reform  and  Terrorism  Prevent  Act  2004  (IRTPA  2004) 

These  four  documents  described  the  existence,  organization  and  management  of  the  Intel¬ 
ligence  Community.  As  directed  in  the  National  Security  Act  of  194760  and  reinforced  by 
EOsl233361,  and  1 3 3 5 5 62 ,  the  DCI  has  oversight  of  the  Intelligence  Community  and  au¬ 
thority  to  develop  policies,  procedures  and  standards  for  activities  of  the  Federal  agencies 
that  fall  within  the  IC.  This  is  separate  and  above  the  DCI’s  responsibilities  for  the  CIA, 
similar  to  SECDEF’s  executive  agent  responsibility  for  NSS.  The  most  recent  document, 
IRTPA  2005,  designated  the  Director  of  National  Intelligence  (DNI)  as  the  head  of  the 
intelligence  community  with  the  authority  to  “...establish  unifonn  security  standards  and 


56  40  USC  subsection  3543,  para  (b)(2) 

57  DoDD  8500.1,  “Information  Assurance”,  24  October  2002. 

58  DoDI  8500.2,  “Information  Assurance  Implementation”,  6  February  2003. 

59  CJCSI  6510. 01D,  “Information  Assurance  and  Computer  Network  Defense”,  15  June  2004. 

60  National  Security  Act  of  1947,  as  amended,  26  July  1947,  “Responsibilities  of  the  Director  of  Central 
Intelligence”,  Sections  103-104. 

61  EO  12333,  “United  States  Intelligence  Activities”,  para  1.5  “Director  of  Central  Intelligence”,  4  Decem¬ 
ber  1981. 

62  EO  13355,  “Strengthened  Management  of  the  Intelligence  Community”,  Section  2,  (b),  27  August  2004. 
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procedures;  and  establish  common  information  technology  standards  protocols,  and  inter¬ 
faces...”63 

D.1.14  OMB  Memo  M-00-07,  28  Feb  200064 

OMB  issued  this  memo  to  provide  guidance  to  Federal  agencies  concerning  incorporating 
and  funding  security  as  part  of  the  agency  information  technology  systems  and  architec¬ 
tures.  It  also  provided  the  criteria  that  would  be  used  to  evaluate  security  for  infonnation 
systems,  as  required  by  CCA  1996.  As  the  revision  to  OMB  Circular  A-130  addressing 
these  details  had  yet  to  be  issued  (in  transmittal  No.  4  described  previously),  this  memo 
was  a  heads-up  to  the  Federal  agencies  on  what  they  should  be  preparing  to  accomplish 
and  reminded  them  of  the  responsibilities  detailed  in  not  only  CCA,  but  also  CSA,  PRA, 
and  the  previous  version  of  the  OMB  Circular  A-130.  Specifically,  the  memo  outlined  the 
principles  and  policy  for  infonnation  security  of  Federal  systems,  including  consistency 
with  security  guidance  issued  by  NIST.  It  also  stated  that  security  for  national  security 
systems  would  be  implemented  in  accordance  with  appropriate  national  security  direc¬ 
tives. 

D.1.15  Government  Information  Security  Reform  (GISR)  2001 65 

This  act  was  the  first  major  revision  of  Federal  infonnation  security  requirements  since 
CCA.  The  statute  created  a  new  subchapter  of  title  44  U.S.  Code,  addressing  the  respon¬ 
sibilities  of  OMB  and  federal  agencies  in  the  area  of  information  security.66  The  purpose 
of  this  subchapter  was  to: 

•  Provide  a  comprehensive  framework  for  establishing  and  ensuring  the  ef¬ 
fectiveness  of  controls  over  infonnation  resources  supporting  Federal  op¬ 
erations  and  assets; 

•  Ensuring  continued  interoperability  of  Federal  Government  systems,  while 
implementing  proved  security  management; 

•  Provide  effective  Government-wide  management  and  oversight  of  the  re¬ 
lated  infonnation  security  risks,  including  coordination  of  information  se¬ 
curity  efforts  throughout  the  civilian,  national  security  and  law  enforce¬ 
ment  communities; 

•  Provide  for  development  and  maintenance  of  minimum  controls  required 
to  protect  Federal  infonnation  and  systems;  and 


63  Intelligence  Reform  and  Terrorism  Prevention  Act  of  2004,  Section  102A.(g), ’’Intelligence  Information 
Sharing”,  17  December  2004. 

64  OMB  Memo  M-00-07,  “Incorporating  and  Funding  Security  in  Information  Systems  Investments”,  28 
Feb  2000. 

65  44  USC  Chapter  35,  subchapter  II,  sections  3531-3536,;  also  Public  Law  106-398,  Section  1061-1065  , 
“Information  Security”,  30  October  2001. 

66  House  Conference  Report  106-945  for  NDAA  2001,  “Government  Information  Security  Reform”,  pp 
852-853. 
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•  Provide  a  mechanism  for  improved  oversight  of  Federal  agency  informa¬ 
tion  security  programs. 

In  addition  to  OMB,  GISR  laid  out  specific  responsibilities  for  the  Secretary  of  Defense 
and  the  Director  of  Central  Intelligence  (DCI),  GSA,  Departments  of  Commerce  and  Jus¬ 
tice,  and  OPM.  Federal  agencies  were  directed  to  appoint  a  senior  information  security 
official  who  would  report  to  the  CIO  and  be  responsible  for  the  execution  of  the  program. 
Aside  from  reemphasizing  the  program  responsibilities  carried  over  from  CCA  1996, 
agencies  were  directed  to  do  an  annual  evaluation  of  their  information  security  programs, 
have  an  independent  audit  of  their  programs,  and  annually  report  the  findings  of  both  the 
internal  evaluation  and  the  independent  audit  to  OMB.  OMB  would  then  compile  these 
reports  into  a  summary  report  to  Congress  annually,  with  the  exception  of  those  from 
DoD  and  the  DCI,  who  would  report  separately  to  the  appropriate  Congressional  commit¬ 
tees.  GISR  contained  a  sunset  provision  for  two  years  from  the  date  of  enactment  (Octo¬ 
ber  2002). 

D.1.16  OMB  Memo  M-01-0867 

OMB  issued  this  memo  to  provide  guidance  on  how  it  expected  Federal  agencies  to  carry 
out  the  requirements  contained  within  GISR  and  clarify  GISR’s  relationship  to  other  ex¬ 
isting  policies  regarding  infonnation  security.  The  primary  focus  of  this  memo,  however, 
was  to  provide  specific  details  on  how  the  Federal  agencies  were  to  comply  with  the  an¬ 
nual  reviews,  including  the  reporting  requirements  and  the  independent  evaluations.  It 
also  described  how  OMB  would  collect  inputs  to  submit  the  consolidated  report  to  Con¬ 
gress. 

D.1.17  Defense-wide  Information  Assurance  Program  (DIAP)  199968 

This  portion  of  the  U.S.  Code  mandated  an  organization  created  by  DoD  in  1998  to  cen¬ 
tralize  the  oversight  for  DoD  of  its  information  assurance  program.  It  described  the  re¬ 
sponsibilities  of  the  program  office,  especially  its  relationship  to  the  DoD  CIO,  the  CCA 
of  1996,  and  national  critical  infonnation  infrastructures.  The  law  also  directed  devel¬ 
opment  of  a  program  strategy  and  submission  of  an  annual  report  with  specific  guidance 
on  the  content  of  that  report.  When  GISR  was  enacted  in  2000,  Congress  clarified  the 
relationship  of  the  DIAP  with  the  requirements  changed  in  GISR  and  described  the  con¬ 
sistency  between  the  annual  report  of  the  DIAP  and  the  report  required  by  GISR.69 

D.1.18  E-Government  Act  of  2002/Federal  Information  Security  Management 
Act  2002  (E-Gov  Act/FISMA) 

Section  3603  of  this  act  mandated  the  Federal  CIO  Council,  and  stated  one  of  its  respon¬ 
sibilities  was  to  work  with  NIST  on  IT  standards,  including  those  for  computer  security. 
Title  III  of  this  act,  the  Federal  Information  Security  Management  Act  of  2002 


67  OMB  Memo  M-01-08,  “Guidance  on  Implementing  the  Government  Information  Security  Reform  Act”, 
16  January  2001. 

68  10  USC  Sec,  2224,  “Defense  Information  Assurance  Program”,  also  Public  Law  106-65,  Section  1043,  5 
Oct  1999. 

69  Public  Law  106-398,  Section  1063,  “Relationship  of  DIAP  to  GISR”,  30  Oct  2000. 
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(FISMA)70,  provided  the  primary  statutory  framework  for  information  assurance  in  the 
Federal  Government  and  replaced  GISR.  It  detailed  the  responsibilities  of  OMB  for  this 
program  and  delegated  certain  of  these  responsibilities  to  the  Secretary  of  Defense  for 
DoD  and  DCI  for  intelligence  systems,  as  well  as  exempting  national  security  systems 
from  OMB ’s  jurisdiction  in  certain  areas.  FISMA  required  each  agency,  including  agen¬ 
cies  with  NSS,  to  develop,  document,  and  implement  agency-wide  information  security 
programs  to  provide  infonnation  security  for  the  infonnation  and  infonnation  systems 
that  support  the  operations  and  assets  of  the  agency,  including  those  provided  or  managed 
by  another  agency,  contractor,  or  other  source.  FISMA  continued  the  annual  reporting 
requirement  begun  by  GISR  and  mandated  the  requirement  for  the  operation  of  the  Fed¬ 
eral  infonnation  security  incident  center  (FEDCIRC). 

D.1.19  OMB  Memos  M-03-1971/M-04-2572 

These  memos  provided  detailed  guidance  to  Federal  agencies  on  the  reporting  require¬ 
ments  to  meet  FISMA  2002  requirements,  incorporating  the  framework  for  annual  IT  se¬ 
curity  reviews,  reporting,  and  remediation  planning  contained  within  the  statute.  The 
memos  also  mandated  quarterly  updates  to  OMB  on  agencies’  IT  security  efforts  using 
quantitative  perfonnance  measures  and  their  progress  in  remediation  of  IT  security  weak¬ 
nesses.  These  measures  are  then  used  in  the  agency’s  E-Gov  scorecard  under  the  Presi- 
dent’s  Management  Agenda.  The  memos  highlighted  the  substantive  changes  from 
GISR  to  FISMA  and  replaced  the  OMB  Memo  M-01-08. 

D.1.20  National  Strategy  for  Homeland  Security  2002  74 

This  strategy  lays  out  the  Administration’s  concept  of  how  it  will  address  the  issue  of 
homeland  security.  As  a  very  high  level  document,  it  provided  little  detail.  It  states  that 
DHS  will  place  a  high  priority  on  protecting  the  U.S.  cyber  infrastructure  and  designates 
DHS  as  the  lead  agency  for  the  Information  and  Telecommunications  sector.  It  called  for 
DHS  to  develop  and  coordinate  the  implementation  of  a  comprehensive  national  plan  to 
protect  the  U.S.  infrastructure.  DHS  was  also  directed  to  provide  a  methodology  for 
identifying  and  prioritizing  critical  assets,  systems  and  functions,  sharing  protection  re¬ 
sponsibility  and  establishing  standards  and  benchmarks.  Information  and  Telecommuni- 


70  Federal  Information  Security  Management  Act,  P.L.  107-347,  §§  301-305,  116  Stat.  2946  (2002).  A 
slightly  different  version  of  the  same  language  was  enacted  as  part  of  the  Flomeland  Security  Act  of  2002, 
P.L.  107-296,  §§  1001-1006,  116  Stat.  2259  (2002). 

71  OMB  Memo  M-03-19,  “Reporting  Instructions  for  the  Federal  Information  Security  Management  Act 
and  Updated  Guidance  on  Quarterly  IT  Security  Reporting”,  6  August  2003. 

72  OMB  Memo  M-04-25,  “FY2004  Reporting  Instructions  for  the  Federal  Information  Security  Manage¬ 
ment  Act”,  23  August  2004. 

73  The  President's  Management  Agenda,  (PMA)  announced  in  the  summer  of  2001,  is  an  aggressive  strat¬ 
egy  for  improving  the  management  of  the  Federal  government.  It  focuses  on  five  areas  of  management 
weakness  across  the  government  where  improvements  and  the  most  progress  can  be  made.  The  web  site  at 
http://www.whitehouse.gov/results/  provides  additional  details. 

74  Office  of  Flomeland  Security,  “National  Strategy  for  Flomeland  Security”,  July  2002,  p.  30-33. 
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cations  was  identified  in  this  Strategy  as  one  of  the  critical  infrastructure  sectors.75  The 
strategy  also  states  that  “while  securing  cyberspace  poses  unique  challenges  and  is¬ 
sues...  our  physical  and  cyber  infrastructures  are  interconnected”  and  called  for  a  Strategy 
to  Secure  Cyberspace. 

D.1.21  National  Strategy  for  Physical  Protection  of  Critical  Infrastructures  and 
Key  Assets  2003 76 

This  document  was  drafted  as  a  follow-on  to  the  National  Strategy  for  Homeland  Security 
partially  in  response  to  the  call  for  a  comprehensive  national  plan.  It  focuses  primarily  on 
the  physical  aspects  of  critical  infrastructures.  Although  telecommunications  is  included 
as  a  sector  in  this  report,  the  cyber  aspects  were  only  addressed  peripherally.  The  primary 
focus  is  on  the  physical  infrastructure  of  facilities,  equipment  (such  as  switches,  access 
tandems  and  others)  connected  by  fiber  and  copper  cable,  and  including  cellular,  micro- 
wave  and  satellite  technologies,  as  well  as  the  wireline  network.77  The  purpose  of  this 
document  was  to  prioritize  and  organize  procedures  to  address  vulnerabilities,  laying  out 
responsibilities  of  the  Federal  government,  state  and  local  governments  and  the  private 
sector. 

70 

D.1.22  National  Strategy  to  Secure  Cyberspace 

Called  for  in  the  National  Strategy  for  Homeland  Security,  this  strategy  directed  Federal 
agencies  to  take  specific  actions  to  improve  the  security  of  Federal  systems.  These  meas¬ 
ures  include: 

•  Continuously  assess  threats  and  vulnerabilities  to  Federal  Cyber  Systems 

•  Agency-specific  processes: 

o  identify  and  document  enterprise  architectures 
o  continuously  assess  threats  and  vulnerabilities 
o  implement  security  controls  and  remediation  efforts 
Additional  challenges  include: 

•  authenticate  and  maintain  authorization  for  users  of  Federal  systems 

•  secure  Federal  wireless  local  area  networks 

•  improved  security  in  government  outsourcing  and  procurement 


75  PDD  63  originally  called  this  sector  “Information  and  Communications”  and  assigned  the  Department  of 
Commerce  as  the  Sector  lead.  The  National  Strategy  for  Homeland  Security  retitled  the  sector  “Informa¬ 
tion  and  Telecommunications”  and  changed  the  sector  lead  designation  to  the  Department  of  Homeland 
Security.  It  includes  what  this  study  calls  Cybersecurity,  as  well  as  Telecommunications,  including  the 
Public-Switched  Telecommunications  Network  (PSTN)  and  physical  infrastructure  thereof. 

76  OHS,  “The  National  Strategy  for  the  Physical  Protection  of  Critical  Infrastructures  and  Key  Assets”, 
February  2003. 

77  Ibid,  “Telecommunications”,  pp;  47-49. 

78  The  White  House,  “The  National  Strategy  to  Secure  Cyberspace”,  February  2003. 
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•  develop  specific  criteria  for  independent  security  reviews  and  reviewers 
and  certification. 

It  recommended  that  the  DHS  use  exercises  to  test  the  security  of  Federal  systems  and  to 
report  the  results  of  those  exercises  to  the  Director  of  OMB.  It  also  directed  DHS  to  work 
with  the  General  Services  Administration  (GSA)  to  develop  an  improved  patch  manage¬ 
ment  system  and  to  ensure  that  agencies  have  made  up-to-date  security  modifications  to 
their  software..  Finally,  it  directed  DHS  to  play  the  central  role  in  implementing  the 
strategy  by  serving  as  the  primary  federal  point  of  contact  (POC)  for  state  and  local  gov¬ 
ernments,  the  private  sector  and  the  American  people  on  issues  related  to  cyberspace  se¬ 
curity.  Of  note,  this  document  called  for  the  review  of  the  NIAP. 

D.1.23  Homeland  Security  Act  2002 79  (HSA  2002) 

This  act,  which  created  DHS,  included  the  creation  of  an  Assistant  Secretary  for  Infra¬ 
structure  Protection  (ASIP),  responsible  for  the  execution  of  the  National  Strategies  for 
Physical  Protection  and  to  Secure  Cyberspace.  In  the  Subsection  titled  “Information  Se¬ 
curity”,  Congress,  among  other  things,  authorized  the  Under  Secretary  for  Information 
Analysis/Infrastructure  Protection  (IAIP)  to  share  information  and  warning  on  threats  and 
vulnerabilities  of  critical  infonnation  systems,  coordinate  response  to  threats  or  attacks  on 
critical  information  systems,  and  provide  technical  assistance  with  respect  to  emergency 
recovery  plans  responding  to  major  failures  of  critical  infonnation  systems  with  state  and 
local  governments,  as  well  as  the  private  sector.  It  brought  together  in  one  organization, 
entities  created  as  a  result  of  PDD  63:  the  National  Infrastructure  Protection  Center 
(NIPC)  from  FBI;  the  Critical  Infrastructure  Assurance  Office  (CIAO)  from  Commerce; 
the  National  Infrastructure  Simulation  and  Analysis  Center  (NISAC)  from  DOE;  the  En¬ 
ergy  Assurance  Office,  also  from  DOE;  and  the  Federal  Computer  Incident  Response 
Center  (FEDCIRC)  from  GSA.  The  act  also  created  “NetGuard”,  a  program  for  volun¬ 
teers  with  critical  skills  in  information  and  communications  technologies  to  assist  in  re¬ 
sponse  these  types  of  emergencies.  Finally,  the  portion  of  the  act  titled  “Computer  Secu¬ 
rity  Enhancement  Act”  amended  sentencing  guidelines  for  certain  computer  crimes,  and 
commissioned  a  study  and  report  on  the  execution  of  this  section. 

D.1.24  Homeland  Security  Presidential  Directive  7:  Critical  Infrastructure  Iden¬ 
tification,  Prioritization  and  Protection80  (HSPD  7) 

This  document  is  the  most  recent  administration  document  establishing  national  policy 
for  Critical  Infrastructure  Protection,  following  the  National  Strategy  for  Homeland  Secu¬ 
rity  and  the  National  Strategy  Physical  Protection  of  Critical  Infrastructures  and  Key  As¬ 
set.  It  officially  supersedes  PDD  63.  HSPD  7  describes  the  following:  (1)  the  policy 
concerning  protection  of  the  nation’s  critical  infrastructure  and  key  resources;  (2)  roles 
and  responsibilities  of  the  Secretary  of  Homeland  Security;  (3)  roles  and  responsibilities 
of  sector-specific  Federal  Agencies;  (4)  roles  and  responsibilities  of  other  Departments, 
Agencies  and  Offices;(5)  collaboration  with  the  private  sector;  and  finally,  (6)  implemen- 


79  Public  Law  107-296,  “Homeland  Security  Act  of  2002”,  Title  II-Information  Analysis  and  Infrastructure 
Protection,  sections  201-205,  25  November  2002. 

so  HSPD  7,  “Critical  Infrastructure  Identification,  Prioritization,  and  Protection”,  17  Dec  2003. 
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tation  details,  including  the  development  of  an  integrated  National  Plan.  This  National 
Plan  would  include  development  of  sector  specific  agency  (SSA)  plans,  with  the  respon¬ 
sibility  for  Information  (Cyber)  assigned  to  the  National  Cyber  Security  Division 
(NCSD)  of  DHS.  Additionally,  Federal  Department  and  agency  heads  were  directed, 
consistent  with  FISMA,  to  identify  and  provide  infonnation  security  protections  for  their 
internal  critical  infrastructure  and  key  resources. 

D.1.25  OMB  Memo  M-04-15^ 

This  memo  provides  implementing  direction  to  Federal  departments  and  agencies  on  the 
formal  submission  of  their  SSA  plans  to  DHS  as  directed  by  HSPD  7.  It  contains  the 
format  to  be  used  by  the  Federal  entities  and  upon  receipt  of  the  plans  by  DHS,  directs  an 
interagency  review,  to  be  coordinated  by  DHS.  Included  within  these  plans  will  be  the 
agency  cybersecurity  plans  that  will  be  reviewed  in  a  manner  consistent  with  reviews  of 
those  reports  submitted  under  FISMA.  The  details  of  these  SSAs  include:  (1)  descrip¬ 
tions  of  existing  capabilities,  including  personnel  and  budget;  (2)  identification  of  the 
process  for  detennining  budget  and  personnel  requirements  for  critical  infrastructure/key 
resources  (CI/KR)  protection,  response,  and  reconstitution  activities;  and  (3)  description 
of  the  process  for  ensuring  independent  oversight  of  CIP  programs.  NSCD,  as  previously 
mentioned,  is  responsible  for  developing  the  SSA  for  cybersecurity,  which  includes  its 
responsibility  to  coordinate  protection,  response  and  reconstitution  activities  with  the 
Federal  agencies  and  the  private  sector  for  cybersecurity.  These  plans  were  due  to  DHS 
by  31  July  2004.  The  most  recent  information  indicates  that  the  plans  (which  were  not 
available  for  review)  will  be  approved  with  the  National  Plan  by  the  Secretary,  DHS,  in 
early  2005. 

D.1.26  Intelligence  Community  (IC)  Policy 

The  primary  document  addressing  this  area  for  the  IC  is  the  Director  of  Central  Intelli- 
gence  Directive  6/3  (DCID  6/3).  This  document  established  the  policy  for  protection  of 
intelligence  information  in  information  systems  and  assigned  responsibility  for  execution 
and  review  of  the  policy.  A  companion  manual  provides  the  detailed  implementation  pol¬ 
icy  as  to  the  expected  methods  of  accomplishing  the  stated  policies.83 

D.1.28  DoD  Policy 

DoD  policies  concerning  information  assurance  (cybersecurity)  are  discussed  in  an  earlier 
section.  The  only  current  DoD  policy  concerning  cybersecurity  as  it  relates  to  Critical 
Infrastructure  Protection  is  DoDD  5160.54.  Titled  “Critical  Asset  Assurance  Program”,  it 


81  OMB  Memo  M-04-15,  “Development  of  Homeland  Security  Presidential  Directive  (HSPD)  7  Critical 
Infrastructure  Protection  Plans  to  Protect  Federal  Critical  Infrastructures  and  Key  Resources”,  17  June 
2004. 

82  DCID  6/3,  “Protecting  Sensitive  Compartmented  Information  Within  Information  Systems”,  5  June 
1999. 

83  DCID  6/3  Manual,  “Protecting  Sensitive  Compartmented  Information  Within  Information  Systems”, 
downloaded  9  November  2004. 
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lays  out  policies  and  responsibilities  for  the  protection  and  assurance  of  DoD  critical  as¬ 
sets  to  support  DoD  missions  worldwide.  84 

D.2  Standards/Guidelines 

In  the  computer  security  environment,  the  establishment  of  standards  is  critical  to  the 
ability  of  an  organization  to  evaluate,  acquire  and  manage  applications,  products  or  ser¬ 
vices.  Congress  recognized  this  need  and  designated  Federal  government  organizational 
responsibilities  for  this  area. 

There  are  a  number  of  government  and  private  sector  entities  that  develop,  promulgate 
and  enforce  or  promote  the  use  of  the  standards.  Standards,  guidelines,  best  practices, 
protocols  and  procedures  are  defined  in  Chapter  2  and  are  used  to  describe  how  individu¬ 
als  in  an  organization  should  accomplish  certain  activities.  The  application  of  a  standard, 
the  most  stringent  of  this  group,  can  be  made  either  mandatory  or  voluntary  by  some  en¬ 
tity  with  the  authority  to  do  so.  An  example  of  a  standard  is  a  set  of  procedures  required 
to  be  developed  by  a  statute  and  promulgated  and  enforced  by  a  Federal  department, 
agency  or  regulating  body  that  has  jurisdiction  over  that  issue.  States  and  local/tribal 
governments  may  enact  state  laws,  ordinances,  or  issue  regulations  or  codes  applicable 
within  their  jurisdictions  that  contain  or  promulgate  standards.  Compliance  normally  has 
some  sort  of  monitoring  and  enforcement  mechanism  and  penalties  to  ensure  that  those 
entities  for  which  the  standard  is  mandatory  do  comply.  State,  local/tribal  governments 
may  adopt  standards  that  are  made  mandatory  for  the  Federal  government  and  make  them 
make  them  applicable  to  those  organizations  under  their  cognizance.  Voluntary  standards 
leave  some  option  to  the  head  of  the  Federal  agency  as  to  whether  to  adopt  in  their  en¬ 
tirety  or  in  part. 

Less  stringent  than  standards  are  guidelines,  which  leave  the  determination  of  how  much 
if  any  of  their  content  to  be  adopted  by  an  organization.  There  can  be  a  number  of  guide¬ 
lines  issued  that  an  organization  may  choose  to  ignore  because  they  are  either  not  appli¬ 
cable  for  their  circumstance  or  their  circumstance  may  require  similar  but  different  pro¬ 
cedures.  Best  practices  are  another  category  of  procedures  that  attempt  to  describe,  for  a 
given  set  of  circumstances,  a  set  of  procedures  that  have  proven  to  be  optimal  for  a  given 
set  of  conditions.  Application  of  best  practices  is  usually  left  up  to  lower  level  managers 
who  are  able  to  assess  the  conditions  to  detennine  which  best  practices,  if  any,  are  useful. 
These  managers  are  also  the  ones  who  would  develop  or  identify  new  best  practices.  Pro¬ 
tocols,  usually  a  given  set  of  procedures  that  should  be  followed  given  a  certain  set  of 
circumstances,  are  also  applicable  in  cybersecurity  and  are  usually  used  to  describe  a  set 
of  technical  procedures  implemented  at  a  software  level. 

D.2.1  National  Institute  of  Standards  and  Technology  Act  1901,  as  amended 
(NIST  Act) 

NIST’s  original  charter  was  laid  out  in  what  was  originally  titled  the  National  Bureau  of 
Standards  Act  (NBSA)  of  1901.  The  National  Bureau  of  Standards  was  created  to 
...’’enhance  the  competitiveness  of  American  industry  while  maintaining  its  traditional 
function  as  lead  national  laboratory  for  providing  the  measurements,  calibrations,  and 


84  DoDD  5160.54,  “Critical  Asset  Assurance  Program  (CAAP)”,  20  January  1998. 
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quality  assurance  techniques  which  underpin  United  States  commerce,  technological  pro¬ 
gress,  improved  product  reliability  and  manufacturing  processes,  and  public  safety;  (2)  to 
assist  private  sector  initiatives  to  capitalize  on  advanced  technology;  (3)  to  advance, 
through  operative  efforts  among  industries,  universities,  and  government  laboratories, 
promising  research  and  development  projects,  which  can  be  optimized  by  the  private  sec¬ 
tor  for  commercial  and  industrial  applications;  and  (4)  to  promote  shared  risks,  acceler¬ 
ated  development,  and  pooling  of  skills  which  will  be  necessary  to  strengthen  America's 
manufacturing  industries...”  .  These  functions  formed  the  foundation  on  which  the 
National  Institute  for  Science  and  Technology  became  a  critical  institution  in  the  devel¬ 
opment  and  promulgation  of  cybersecurity  standards  and  procedures  as  directed  in  other, 
more  recent  statutes86 

D.2.2  CSA 1987 

This  statute  amended  the  NIST  Act  by  establishing  a  computer  standards  program  and 
assigned  NIST  the  responsibility  to  develop  standards  and  guidelines  for  Federal  com¬ 
puter  systems  on  the  security  and  privacy  of  sensitive  information.  Specifically  NIST  was 
responsible  for: 

•  Developing  standards,  guidelines  and  associated  methods  and  techniques 
for  computer  systems; 

•  Except  for  NSS,  develop  uniform  standards  and  guidelines  for  Federal 
computer  systems; 

•  Develop  technical,  management,  physical  and  administrative  standards 
and  guidelines  for  security  and  privacy  of  sensitive  information  in  Federal 
computer  systems; 

•  Submit  standards  and  guidelines,  along  with  recommendations  as  to  which 
should  be  made  compulsory  and  binding,  to  the  Secretary  of  Commerce 
for  promulgation  to  Federal  agencies; 

•  Develop  guidelines  training  for  operators  in  security  awareness  and  ac¬ 
cepted  security  practice; 

•  Develop  validation  procedures  for,  and  evaluate  the  effectiveness  of  stan¬ 
dards  and  guidelines  promulgated  under  the  CSA  through  research  and  li¬ 
aison  with  other  government  and  private  agencies; 

•  NIST  was  also  supposed  to  draw  on  the  work  done  by  the  NSA  for  the  na¬ 
tional  security  community  where  such  work  is  consistent  with  its  other  ef¬ 
forts  for  protecting  sensitive  information  in  Federal  computer  systems. 

The  Secretary  of  Commerce  was  authorized  to  promulgate  standards  and  guidelines  per¬ 
taining  to  Federal  computer  systems  and  to  make  such  standards  compulsory  and  binding 
to  the  extent  deemed  necessary.  Only  the  President  could  disapprove  or  modify  these 


8515  USC  Chapter  7,  section  271, .  National  Institute  of  Standards  and  Technology,  Aug.  23,  1988  . 
86  15  USC  section  278g-3,  “NIST”,  Computer  Standards  Program,  25  November  2002. 
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standards.  The  statute  provided  a  process,  however,  by  which  these  computer  security 
standards  could  be  waived  by  the  Secretary  of  Commerce  if  it  was  detennined  that  com¬ 
pliance  would  adversely  affect  the  accomplishment  of  the  mission  or  cause  a  major  finan¬ 
cial  impact  not  offset  by  Government-wide  savings.  The  Secretary  could  delegate  to  the 
head  of  Federal  agencies  the  authority  to  waive  such  standards  and  who  could  then  fur¬ 
ther  delegate  this  authority  to  certain  senior  officials  of  the  agency. 

D.2.3  National  Technology  Transfer  and  Advancement  Act  of  1995  (NTTAA 
1995) 87 

The  significance  of  this  Act  for  cybersecurity  for  the  Federal  Government  is  that  it  did 
two  things:  (1)  assigned  NIST  the  responsibility  for  comparing  standards  used  in  a  vari¬ 
ety  of  venues  with  the  standards  adopted  or  recognized  by  the  Federal  Government  and 
coordinating  the  use  of  private  sector  standards  by  Federal  agencies;88  and  (2)  required 
all  Federal  agencies  and  departments  to  use  technical  standards  developed  or  adopted  by 
voluntary  consensus  standards  bodies,  except  where  inconsistent  with  applicable  law  or 
otherwise  impractical.  Federal  agencies  are  encouraged  to  participate  in  these  stan¬ 
dards  bodies  to  the  extent  practicable  to  ensure  that  their  concerns  are  addressed  in  the 
development  of  the  voluntary  consensus  standard  by  the  standards  body.  The  signifi¬ 
cance  of  this  Act  is  that,  where  no  Federal  or  agency  standard  exists  for  Federal  agencies 
to  implement  and  a  voluntary  consensus  standard  exists,  the  adoption  of  this  voluntary 
consensus  standard  becomes  mandatory  by  Federal  agencies. 

D.2.4  OMB  Circular  A-119,  Federal  Participation  in  the  Development  and  Use 
of  Voluntary  Consensus  Standards  and  in  Conformity  Activities,  10  Feb  1998 

This  Circular  provides  implementing  direction  to  Federal  Agencies  for  the  NTTAA  of 
1995.  It  clarifies  the  applicability  of  this  policy  and  defines  some  of  the  terms  found  in 
the  Act.  “Voluntary  consensus  standards  are  defined  as  “those  standards  developed  or 
adopted  by  voluntary  consensus  standards  bodies,  both  domestic  and  international”.90 
The  Circular  restates  the  policy  requirement  for  Federal  agency  adoption  of  voluntary 
consensus  standards  and  established  a  annual  reporting  requirement  for  Federal  agencies 
on  decisions  to  use  government-unique  standards  in  lieu  of  voluntary  consensus  stan¬ 
dards.  91  Although  not  specific  to  cybersecurity,  this  Circular  has  implications  for  the 
Standards  development  and  implementation  for  Federal  agencies. 


87  Public  Law  104-113,  “National  Technology  Transfer  and  Advancement  Act  of  1995”,  7  March  1996. 

88  Ibid,  Section  12,  para  (a)(3). 

89  Ibid,  Section  12,  para  (d)  (1)  and  (d)(3). 

90  OMB  Circular  A-119,  “Federal  Participation  in  the  Development  and  Use  of  Voluntary  Consensus  Stan¬ 
dards  and  in  Conformity  Assessment  Activities”,  10  February  1998,  para  4. a. 

91  Ibid,  para  9. 


D-16 


D.2.5  15  CFR  Part  287,  Guidance  on  Federal  Conformity  Assess¬ 

ment  Activities,  10  August  2000. 92 

This  document  provides  policy  guidance  to  Federal  agencies  on  the  evaluation  of  con¬ 
formity  assessment  activities  relative  to  NTTAA  1995.  “Conformity  assessment”  is  de¬ 
fined  as  any  activity  concerned  with  determining  directly  or  indirectly  that  requirements 
are  fulfilled  or  requirements  for  products,  services,  systems,  and  organizations  defined  by 
law  or  regulation  or  by  an  agency  in  a  procurement  activity  and  includes  accreditation, 
certification  and  inspection.9' 

D.2.6  CCA  1996 

This  act  made  some  significant  changes  in  the  Standards  area,  beginning  with  giving  the 
Director  of  OMB  oversight  responsibility  for  the  development  and  implementation  of 
standards  and  guidelines  developed  by  the  Secretary  of  Commerce,  through  NIST,  as  re¬ 
quired  by  the  CSA  of  1987. 94  As  this  act  repealed  the  section  of  the  CSA  concerning  the 
authority  of  the  Secretary  of  Commerce  to  make  certain  standards  and  guidelines  com¬ 
pulsory  and  binding  for  Federal  agencies,  CCA  clarified  that  authority,  including  the  abil¬ 
ity  to  waive  standards.  It  was  specified  that  the  authority  to  waive  standards  could  only  be 
delegated  as  far  as  the  CIOs  of  organizations  identified  within  the  act.  EO  13011  was 
issued  by  the  Administration  in  1996  as  described  in  an  earlier  section,  and  established  a 
Government  Information  Technology  Services  Board.  This  Board,  among  other  things, 
was  responsible  for  making  recommendations  and  assisting  with  developing,  with  NIST 
and  established  standards  bodies,  standards  and  guidelines  pertaining  to  Federal  informa¬ 
tion  systems.95  The  responsibilities  of  the  Secretary  of  Commerce  were  reiterated,  with 
the  additional  duty  of  taking  into  consideration  the  recommendations  of  the  agencies,  the 
CIO  Council,  and  the  aforementioned  Services  Board.96 

D.2.7  OMB  Circular  A-130 

This  circular  reiterated  the  responsibilities  of  both  Federal  agencies,  in  general,  and  the 
Department  of  Commerce,  in  specific,  regarding  the  use  and  development  of  standards. 
OMB  directed  Federal  agencies  to  use  voluntary  standards  and  Federal  Information  Proc¬ 
essing  Standards  (FIPS)  (promulgated  by  NIST)  where  appropriate  or  required.  Agencies 
were  required  to  implement  policies,  standards  and  procedures  issued  by  OMB,  the  De¬ 
partment  of  Commerce,  GSA  and  OPM.  Agencies  were  also  responsible  for  incorporat¬ 
ing  the  more  stringent  standards  for  national  security  systems  as  appropriate.  OMB  di¬ 
rected  the  Secretary  of  Commerce  to  develop  and  issue  FIPS  and  guidelines  to  ensure  ef¬ 
ficient  and  effective  security  of  information  technology,  taking  into  consideration  rec¬ 
ommendations  of  agencies  and  the  Federal  CIO  Council. 


92  15  CFR  Part  287,  “Guidance  on  Federal  Conformity  Assessment  Activities”,  10  August  2000. 

93  Ibid,  Section  287.2. 

94  ITMRA  Title  III,  Subtitle  A,  Section  5112  (d). 

95  EO  13011  Sec.  4(a)(4). 

96  EO  13011  Sec  8. 


D-17 


D.2.8  FISMA  2002 

This  statute  made  substantial  changes  in  the  standards  area  for  IT  and  IT  security.  In 
general,  heads  of  Federal  agencies  were  required  to  comply  with  information  security 
standards  promulgated  under  this  act  and  information  system  security  standards  for  na¬ 
tional  security  systems.  The  Secretary  of  Commerce  was  given  the  authority  to  make 
mandatory  the  standards  and  guidelines  developed  by  NIST  and,  further,  were  required  to 
act  on  any  standards/guidelines  submitted  by  NIST  within  6  months  of  receipt.  These 
mandatory  standards  and  guidelines  could  only  be  modified  or  disapproved  by  the  Presi¬ 
dent.  Heads  of  Federal  agencies  were  allowed  apply  more  stringent  standards  and  guide- 

Q7 

lines,  but  must  include  these  mandatory  standards  and  guidelines. 

The  Director  of  OMB  was  responsible  for  providing  oversight  of  the  development  of 
standards  and  guidelines  for  information  security  and  ensuring  that  these  standards  are 
coordinated  with  those  entities  responsible  for  developing  standards  and  guidelines  for 
national  security  systems  to  ensure  they  are  complementary. 98 

FISMA  also  provided  significant  direction  to  NIST  directly,  on  the  standards  to  be  devel¬ 
oped,  including  a  definition  of  what  constituted  minimum  security  standards.  NIST  was 
directed  to  work  with  NSA  to  ensure  that  standards  developed  for  Federal  agencies  were 
consistent  with  those  developed  for  NSS.  Additionally,  NIST  was  directed  to  submit 
those  standards  that  should  be  made  mandatory  to  the  Secretary  of  Commerce  within  12 
months,  taking  into  consideration  any  recommendations  from  the  Information  and  Secu¬ 
rity  Privacy  Board99.  This  Board,  fonnerly  called  the  Computer  System  Security  and 
Privacy  Advisory  Board,  was  given  the  charter  to  advise  the  Secretary  of  Commerce,  and 
Director,  OMB,  on  a  number  of  information  security  matters,  including  reviewing  and 
recommending  proposed  standards  and  guidelines. 100  This  statute  also  repealed  the  sec¬ 
tion  of  the  CSA,  which  had  authorized  the  Secretary  of  Commerce  to  waive  standards 
that  were  made  compulsory  and  binding,  with  the  ability  to  further  delegate  that  waiver 
authority  to  senior  officials  of  Federal  agencies.101 

D.2.9  OMB  Memo  M-03-19102 

In  this  memo,  OMB  outlines  the  changes  in  policy  brought  about  by  FISMA  and  provides 
additional  policy  direction  to  Federal  agencies  as  a  result  of  that  statute.  The  memo  reit¬ 
erates  the  direction  to  NIST  concerning  compulsory  and  binding  standards  and  encour¬ 
ages  agencies  to  participate  in  the  drafting  and  review  of  these  drafts.  OMB  will  issue,  as 
needed,  accompanying  implementing  guidance  for  the  NIST  standards  and  guidelines. 


97  Section  11331,  title  40  US  Code,  “Responsibilities  for  Federal  Information  Systems  Standards” 

98  Sec  301,  subsection  3543 

99  Section  303  FISMA,  which  amended  section  20  of  the  NIST  Act  (15  USC  278g-3) 

100  Section  302  FISMA,  which  amended  section  21  of  the  NIST  Act  (15  USC  278g-4) 

101  Section  303  FISMA,  which  amended  section  20  of  the  NIST  Act  (15  USC  278g-3) 

102  OMB  Memo  M-03-19,  “Reporting  Instructions  for  the  Federal  Information  Security  Management  Act 
and  Updated  Guidance  on  Quarterly  IT  Security  Reporting”,  OMB,  Washington,  D.C.  6  August  2003. 
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D.2.10  NIST  Standards  and  Guidelines 


The  mandatory  Federal  Information  Processing  Standards  (FIPS)  issued  by  NIST  primar¬ 
ily  deal  with  various  encryption  schemes,  although  that  situation  is  changing  as  NIST  fi¬ 
nalizes  the  coordination  of  more  FIPS  to  meet  the  minimum  set  required  by  FISMA  2002. 
The  more  widely  applicable  guidance  for  security  procedures  and  incident  handling  are 
addressed  in  Special  Publications  (SP)  that  are  voluntary  and  may  be  selectively,  imple¬ 
mented,  if  at  all.  OMB  has  made  only  one  SP  mandatory,  but  strongly  encourages  Fed¬ 
eral  agency  heads  to  comply  to  the  maximum  extent  possible,  with  other  applicable  SP. 

As  a  result  of  the  tasking  to  NIST  regarding  standards  and  guidelines,  NIST  has  issued  a 
number  of  FIPS  (Federal  Information  Processing  Standards)  and  SP  (Special  Pubs)  ad¬ 
dressing  aspects  of  information  security.  Some  of  these  documents  are  mandatory,  while 
others  are  voluntary.  A  list  of  these  documents  is  provided  at  Table  1  in  Chapter  3.  Cop¬ 
ies  of  the  majority  of  the  documents  are  available  at  the  following  url: 
http://csrc.nist.gov/publications/index.html. 

D.2.11  National  Security  Standards  &  Guidelines 

National  security  systems  are  exempt  from  the  standards  and  guidelines  promulgated  by 
NIST  for  the  Federal  government.  Standards  and  guidelines  for  national  security  systems 
are  developed,  prescribed,  enforced  and  overseen  as  authorized  by  law  and  as  directed  by 
the  President. 103  As  the  Executive  Agent  for  NSS,  SECDEF  approved  and  provides 
minimum  security  standards  and  doctrine  for  NSS.  DIRNSA,  as  the  National  Manager 
for  NSS,  is  responsible  form  reviewing  and  approving  all  standards,  techniques,  systems 
and  equipment  related  to  the  security  of  national  security  systems  and  coordinating  with 
NIST  as  required  by  the  CSA  of  1987  (as  modified  by  FISMA).  FISMA  requires  Federal 
agencies  with  national  security  systems  to  comply  with  standards  and  guidelines  devel¬ 
oped  for  NSS  as  described  above.  CNSS  has  issued  detailed  policy  and  guidance  to  the 
national  security  community  under  its  authority  per  NSD  42.  104 . 

D.2.12  Department  of  Defense  Standards  &  Guidelines 

The  authority  to  develop  standards  for  DoD  was  delegated  to  Assistant  Secretary  of  De¬ 
fense  for  Network  Infrastructure  Interoperability  (ASD(NII))/DoD  CIO  for  information 
assurance  standards,  in  cooperation  with  NSA,  and  to  the  Defense  Information  Systems 
Agency  (DISA)  for  IT  standards.  DoD  has  long  recognized  the  need  for  standards  and 
guidelines  to  manage  its  IT  infrastructure,  and  in  particular,  information  secu¬ 
rity/assurance.  The  primary  policy  document  for  the  IT  infrastructure  is  DoDD  8100.1 
Global  Information  Grid  Overarching  Policy105  that  implements  the  requirements  of  the 
Clinger-Cohen  Act  for  DoD.  The  document  addressing  standards  and  guidelines  for  in- 


104  40  USC  ssl  133 1,  para  (a)(2)) 

104  The  complete  list  of  those  documents  is  found  in  the  Index  of  National  Security  Systems  Issuances, 
dated  October  2004  and  is  maintained  by  the  CNSS  Secretariat,  located  at  NSA.  This  index  can  be  found  at 
the  following  url:  http://www.nstissc.gov/html/library.html. 

105  DoD  Directive  8100.1,  “Global  Information  Grid  (GIG)  Overarching  Policy”,  19  September  2002. 
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formation  security/assurance  is  DoDD  8500.1 106  and  its  companion  Instruction  DoDI 
8500.2,  ,  already  mentioned  in  the  cybersecurity  theme.  Heads  of  DoD  components  are 

responsible  and  accountable  for  implementing  these  standards  within  the  parts  of  the 
DoD  IT  infrastructure  under  their  operation  and  control  and  may  incorporate  more  strin¬ 
gent  standards,  as  long  as  interoperability  across  the  DoD  IT  infrastructure  is  not  im¬ 
pacted. 

D.2.13  Intelligence  Community  Standards  &  Guidelines 

As  directed  in  the  National  Security  Act  of  1947 108  and  reinforced  by  EOsl2333109,  and 
13355  u0,  the  DCI  has  oversight  of  the  Intelligence  Community  and  authority  to  develop 
policies,  procedures  and  standards  for  activities  of  the  Federal  agencies  that  fall  within 
the  IC.  This  is  separate  and  above  the  DCI’s  responsibilities  for  the  CIA,  similar  to 
SECDEF’s  executive  agent  responsibility  for  NSS.  The  most  recent  document,  IRTPA 
2005,  designated  the  Director  of  National  Intelligence  (DNI)  as  the  head  of  the  intelli¬ 
gence  community  with  the  authority  to  “...establish  unifonn  security  standards  and  pro¬ 
cedures;  and  establish  common  infonnation  technology  standards  protocols,  and  inter¬ 
faces...”111 

Two  primary  documents  addressing  IC  concerns  with  regard  to  protecting  sensitive  com- 
partmented  infonnation  (SCI)  within  infonnation  systems:  DCI  Directive  6/3  Policy  and 
its  accompanying  Manual.  As  mentioned  previously,  it  is  in  these  documents  that  the 
DCI  lays  out  the  standards  and  guidelines  regarding  information  security  in  those  systems 
of  interest.  Federal  agencies  having  components  designated  as  part  of  the  IC  must  imple¬ 
ment  these  requirements. 

National  Security  Act  of  1947  (NSA  1947) 

Gives  the  DCI  authority  to  develop  policies,  procedures  and  standards  for  activities  of  the 
Federal  agencies  that  fall  within  the  Intelligence  Community. 


106  DoD  Directive  8500.1,  “Information  Assurance”,  24  October  2002. 

107  DoD  Instruction  8500.2,  “Information  Assurance  (IA)  Implementation”,  6  February  2003. 

I0fi  National  Security  Act  of  1947,  as  amended,  26  July  1947,  “Responsibilities  of  the  Director  of  Central 
Intelligence”,  Sections  103-104. 

109  EO  12333,  “United  States  Intelligence  Activities”,  para  1.5  “Director  of  Central  Intelligence”,  4  Decem¬ 
ber  1981. 

110  EO  13355,  “Strengthened  Management  of  the  Intelligence  Community”,  Section  2,  (b),  27  August  2004. 

111  Intelligence  Reform  and  Terrorism  Prevention  Act  of  2004,  Section  102 A.  (g), ’’Intelligence  Information 
Sharing”,  17  December  2004. 

112  Director  of  Central  Intelligence  Directive  6/3,  “Protecting  Sensitive  Compartmented  Information  within 
Information  Systems”,  5  June  1999. 
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Intelligence  Reform  Act  of  2004  (IRA  2004) 

Gives  the  Director  of  National  Intelligence  (DNI),  as  head  of  the  intelligence  community, 
authority  to  “...establish  uniform  security  standards  and  procedures;  and  establish  com- 

ill 

mon  information  technology  standards  protocols,  and  interfaces. . .” 

Collectively,  these  statutes  established  the  requirement  for  standards  and  guidelines  for 
the  Federal  government  and  assigned  the  responsibility  for  developing  and  promulgating 
these  standards  and  guidelines  either  to  NIST,  the  Secretary  of  Defense,  or  the  head  of  the 
Intelligence  Community. 

D.2.14  National  and  International  Standards  Bodies 

The  discussion  of  standards  would  be  incomplete  if  it  did  not  include  the  existence  of  na¬ 
tional  and  international  standards  bodies,  outside  of  the  Federal  government.  These  bod¬ 
ies  issue  standards  that  have  been  developed  by  technical  committees  representing  the 
private  sector,  academia  and  government  activities.  NIST,  DoD,  NSA  and  a  number  of 
Federal  government  organizations  participate  on  a  regular  basis  in  developing  these  stan¬ 
dards,  particularly  those  having  a  specific  application  in  their  area  of  interest  or  responsi¬ 
bility.  These  standards  are  voluntary  unless  adopted  by  an  oversight  organization  with 
the  authority  to  enforce  implementation.  Examples  include:  American  National  Stan¬ 
dards  Institute  (ANSI),  Institute  for  Electrical  and  Electronics  Engineers  (IEEE),  Interna¬ 
tional  Telecommunications  Union  (ITU),  Internet  Engineering  Task  Force  (IETF)  and 
others.  An  example  of  a  standard  developed  by  one  of  these  bodies  is  ISO/IEC  17799 
that  provides  an  international  agree-upon  standard  for  information  security  manage¬ 
ment. 114 

D.2.15  Best  Practices 

The  most  flexible  category  of  the  standards/guidelines  area  is  that  of  “best  practices”. 
“Best  practices”  refer  to  strategies,  policies,  procedures  and  other  action-related  elements 
of  cybersecurity  that  are  generally  accepted  as  being  the  most  successful  or  cost-effective. 
Many  organizations  sponsor  or  encourage  the  development  of  best  practices,  partially  in 
an  effort  to  get  input  from  disparate  organizations  within  government  and  in  the  private 
sector,  but  also  to  build  consensus  on  general  practices  without  resorting  to  issuance  of 
policies  and  regulations.115  NIST’s  Computer  Security  Division  (CSD)  hosts  the  Federal 
Agency  Security  Practices  (FASP)  for  the  Federal  CIO  Council  that  provides  a  mecha¬ 
nism  for  information  sharing  and  collaboration  for  Federal  security  professionals.  It  also 
contains  links  to  a  number  of  public  and  private  sector  best  practices  sites  for  additional 
information. 116  “Best  Practices”  can  provide  useful  information  to  assist  these  profes- 


113  Intelligence  Reform  Act  of  2004,  Section  102A.(g), ’’Intelligence  Information  Sharing”,  17  December 
2004. 

114  ISO/IEC  17799,  “Information  Technology:  Code  of  Practice  for  Information  Security  Management”, 
First  Edition  2000-12-01. 

115  The  FCC’s  Network  Reliability  &  Interoperability  Council  is  one  example  of  a  regulatory  advisory 
group  sponsoring  best  practices  in  telecommunications/Cybersecurity  to  prevent  having  to  issue  controver¬ 
sial  and  expensive  regulations.  Web  site  is  found  at  http://www.bell-labs.com/cgi-user/krauscher/bestp.pl. 

116  Web  site  address  is:  http://csrc.nist.gov/fasp/. 
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sionals,  but  the  voluntary  nature  and  varying  technical  detail  can  prove  problematic  in 

1 17 

implementation  and  consistency. 

D.2.16  Applicability  of  Requirements 

One  of  the  difficulties  in  policy  implementation  is  that  Federal  agencies  are  required  to 
identify  the  community  of  interest  in  which  each  IT  system  belongs.  Table  20  illustrates 
how  Federal  agencies  IT  systems  may  be  identified  with  multiple  communities  and  be 
subject  to  various  requirements.  It  is  possible  that  a  single  Department  may  have  entities 
and  systems  belonging  to  all  four  communities  of  interest  and  must,  therefore,  implement 
four  sets  of  policies.  The  DoD,  for  example,  has  systems  that  fall  in  four  communities  of 
interest.  To  simplify  the  issue,  Departments  or  Agencies  have  the  authority  to  adopt  more 
stringent  policies  and  can  thereby  reduce  the  implementation  complexity  by  bringing  all 
of  their  systems  under  one  or  two  sets  of  policies.  A  case  in  point,  once  again,  is  the 
DoD.  NSTISSP  11  requires  all  computer  security  products  introduced  into  National  Se¬ 
curity  Systems  be  certified  through  the  NIAP  process.  National  Security  Systems  explic¬ 
itly  exclude  systems  used  for  routine  administrative  and  business  applications  (including 
payroll,  finance,  logistics,  and  personnel  management  applications).  As  a  result,  if  a  De¬ 
partment  or  Agency  adopts  these  requirements  throughout  its  respective  organization, 
some  systems  owned  or  operated  by  or  on  behalf  of  these  organizations  may  nonetheless 
not  be  covered  by  this  requirement.  In  DoD’s  case,  DoDD  8500.1  [DoD2002a]  requires 
all  DoD  systems  meet  NSTISSP  11  requirements,  eliminating  different  standards  for  Na¬ 
tional  Security  Systems  and  the  rest  of  the  systems  in  DoD.  For  Federal  agencies  that  do 
not  adopt  a  single  set  of  standards,  the  challenge  becomes  which  set  of  standards  applies 
to  which  IT  systems  in  a  consistent,  coherent  way,  at  the  same  time  ensuring  that  the  ap¬ 
plication  of  these  standards  does  not  result  in  non-interoperable  systems. 


117  A  good  discussion  of  the  limitations  of  the  use  of  best  practices  is  contained  in  the  following  report: 
Fischer,  Eric  A,  “Creating  a  National  Framework  for  Cybersecurity:  An  Analysis  of  Issues  and  Options”, 
RL3277,  22  Feb  2002,  page  38. 
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Table  20.  Federal  Departments  and  Agencies  and  Communities 


Federal  Departments  and  Agencies 

/  c? 

/ 

/ 

/  o° 

■A  / 

?  /  -S'  .£• 

/  -§  A 
/  A  A  a 

/ 

^  4 

/  3? 

/  ,0 

/  o 

/ 

f  / 

CIA 

DDCI/CM 

Defense 

Air  Force 

AF  Intel. 

Army 

Army  Intel. 

Navy 

Navy  Intel. 

Marine  Corp 

DIA 

JCS 

MC  Intel. 

NGA 

NRO 

NSA 

DISA 

DTRA 

DLA 

State 

T  reasury 

Energy 

Justice 

Homeland  Security 

Coast  Guard 

CG  Intel. 

FEMA 

_  [ncs 

Commerce 

_ CIAO 

Health  and  Human  Services 

T  ransportation 

GSA 

OSTP  _[  _ 

Agriculture 

Education 

Housing  and  Urban  Development 

Interior 

Labor 

Veterans  Affairs 

D.3  Education,  Training  &  Awareness 

Education,  training  and  awareness  (ET&A)  are  critical  components  of  any  information 
security  program.  Requirements  for  these  activities  are  contained  in  numerous  policy 
documents,  along  with  specific  assignments  of  responsibility. 
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D.3.1  CSA  1987/NIST  Act 

The  CSA,  for  the  first  time,  required  mandatory  periodic  computer  security  training  of  all 
Federal  personnel  involved  in  the  management,  use  or  operation  of  Federal  computer  sys- 
terns  containing  sensitive  information.  As  modified  by  CSA,  NIST  was  given  two 
tasks  that  focus  on  training:  (1)  develop  guidelines  for  use  by  Federal  personnel  in  train¬ 
ing  their  employees  in  security  awareness  and  accepted  security  practice;  and  (2)  assist¬ 
ing  the  Office  of  Personnel  Management  (OPM)  in  developing  regulations  pertaining  to 
training  in  this  area.  Further  detail  is  provided  in  the  Act  on  the  objectives  of  this  train¬ 
ing.  Additionally,  OPM  was  directed  to  issue  regulations  concerning  the  scope  and  man¬ 
ner  of  this  training. 119 

D.3.2  CCA  1996 

This  statute  contained  additional  tasking  regarding  education  and  training.  First,  it  as¬ 
signed  the  responsibility  for  monitoring  the  status  of  training  of  Federal  personnel  to  the 
Director,  OMB.  Additionally,  Chief  Information  Officers  (CIO)  of  Federal  agencies,  were 
responsible  for  training  their  IT  personnel.  The  training  cited  was  not  specific  to  infor¬ 
mation  security,  but  concerned  information  technology  generally  as  follows: 

“...The  Director  [OMB]  shall  monitor  the  development  and  implementation  of 

120 

training  in  information  resources  management  in  executive  agency  personnel...” 

CIO’s  of  agencies  shall  “...in  order  to  rectify  any  deficiency  in  meeting  those  re¬ 
quirements,  develop  strategies  and  specific  plans  for  hiring,  training,  and  professional 
121 

development...” 

It  is  clear  that  the  statute  intended  for  the  CIOs  of  Federal  agencies  to  ensure  that  they 
had  sufficient  numbers  of  trained  personnel  to  carry  out  the  infonnation  technology  re¬ 
quirements,  including  that  of  information  security. 

D.3.3  EO  13011  Federal  Information  Technology 

This  EO  was  the  initial  Executive  Branch  document  to  implement  the  requirements  of  the 
CCA.  It  laid  out  the  responsibilities  of  Agency  Heads  and  included  support  for  appropri¬ 
ate  training  of  personnel  in  this  area.  It  also  established  the  Chief  Information  Officer 
Council  (Federal  CIO  Council)  and  assigned  that  body  the  responsibility  for  assessing 

and  addressing  the  hiring,  training,  classification,  and  professional  development  needs  for 

122 

IT  management. 

D.3.4  OMB  Circular  A-130 

This  document  repeated  the  training  requirements  stated  in  CCA.  OPM  was  specifically 
charged  with  developing  and  conducting  training  programs  for  Federal  personnel  on  in- 


118  CSA,  Section  2,(b)(4). 

119  Ibid,  Section  5,  “Federal  Computer  System  Training”. 

120  CCA,  Pub  Law  104-106,  Section  5112,  para  (i). 

121  Ibid,  Section  5125,  para  (c)(3)(C). 

122  EO  13011,  “Federal  Information  Technology”,  16  July  1996,  Sections  2  and  3. 
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formation  resources  management;  evaluating  future  personnel  management  and  staffing 
in  this  area,  and  developing  training  programs  for  Federal  personnel  in  the  design,  opera- 
tion  or  maintenance  of  information  systems.  The  Circular  Appendix  on  Security  pro¬ 
vides  additional  details  on  the  required  training.  Specifically,  agencies  are  required  to 
implement  and  maintain  automated  information  security  programs  that,  among  other 
things,  ensure  that  all  individuals  are  trained  in  how  to  fulfill  their  security  responsibili¬ 
ties  before  allowing  them  access  to  the  system,  demonstrate  appropriate  behavior  while 
on  the  systems;  and  receive  periodic  refresher  training  for  continued  access  to  systems.124 
Specialized  training  for  personnel  is  required  for  major  applications. 125  OPM  has  two 
additional  responsibilities  in  information  security  training  (1)  ensuring  that  its  regulations 
concerning  computer  security  training  are  effective;  and  (2)  assisting  the  Department  of 
Commerce  in  updating  and  maintaining  guidelines  for  training  in  computer  security 
awareness  and  accepted  computer  security  practice. 126 

D.3.5  FISMA 

This  statute  took  security  training  requirements  laid  out  in  the  CSA  and  expanded  them. 
Agencies  CIOs  were  tasked  with  ensuring  that  IT  personnel  with  significant  security  re- 
sponsibilities  were  trained.  The  CIOs  must  also  ensure  they  have  sufficient  numbers  of 
trained  personnel  to  handle  their  designated  responsibilities.  "  Additionally,  agencies,  as 
part  of  their  information  security  program,  must  conduct  security  awareness  training  for 
all  personnel,  including  contractors,  of  both  the  information  security  risks  associated  with 
their  responsibilities  and  their  responsibility  to  comply  with  agency  policies  and  proce¬ 
dures  intended  to  reduce  these  risks.129  Finally,  this  training  must  be  documented  in  the 

130 

performance  plan  that  agency  CIOs  submit  to  OMB. 

D.3.6  OMB  Memo  M-03-19 

This  memo  provided  additional  guidance  to  the  Federal  agencies  on  reporting  require¬ 
ments  for  FISMA.  That  guidance  also  included  a  requirement  to  report  on  the  status  of 
security  training  and  awareness  of  agency  employees  (including  contractors).  131 


123  OMB  Circular  A-130  ,  Section  9.f. 

124  Ibid,  Appendix  III,  “Security  of  Federal  Automated  Information  Resources”,  para  A.3.a,20,  b. 

125  Ibid,  para  A.3.b.2.b. 

126  Ibid,  para  A.4.e. 

127  FISMA,  44  USC  35,  section  3544,  para  (a)(3)(D).  However,  clarification  of  what  Congress  or  OMB 
meant  by  significant  is  not  defined,  leaving  some  confusion  as  to  the  distinction  between  significant  and 
non-significant  for  the  purposes  of  this  requirement. 

128  Ibid,  para  (a)(4). 

129  Ibid,  para  (b)(4). 

130  Ibid,  para  (d)(1)(B). 

131  OMB  Memo  M-03-19,  “Reporting  Instructions  for  the  Federal  Information  Security  Management  Act 
and  Updated  Guidance  on  Quarterly  IT  Security  Reporting”,  6  August  2003. 
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D.3.7  OPM  Personnel  Regulations  for  Federal  personnel 

OPM  issued  its  regulation  prescribing  mandatory  information  security  training  for  Fed¬ 
eral  personnel,  originally  in  1991,  and  revised  it  in  2004  to  bring  the  regulation  into  com- 
pliance  with  FISMA  .  The  regulation  directs  Executive  Agencies  to  develop  informa¬ 
tion  systems  security  awareness  training  plans,  including  specific  training  targeted  at  in¬ 
dividuals  identified  as  having  significant  information  security  responsibilities.  All  users 
are  required  to  have  initial  training  prior  to  being  allowed  access  to  systems,  annual 
awareness  training,  and  additional  training  when  there  is  a  significant  change  in  responsi¬ 
bilities.  Standards  and  guidelines  for  training  are  provided  through  NIST  and  are  de¬ 
scribed  in  the  next  paragraph. 

D.3.8  NIST  Standards  and  Guidelines  for  Training  for  Federal  personnel 

As  directed  by  CSA,  NIST  developed  standards  and  guidelines  for  information  security 
training  for  Federal  agencies.  It  is  contained  in  two  documents,  NIST  SP  800-16  and 

i 

800-50.  "  The  learning  continuum  modeled  in  this  guidance  provides  the  relationship 
between  awareness,  training,  and  education.  The  first  document  contains  a  methodology 
that  can  be  used  to  develop  training  courses  for  a  number  of  audiences,  which  may  have 
significant  information  security  responsibilities.  The  second  document  is  intended  to  aid 
Federal  agencies  in  developing  and  information  security  FISMA  compliant  ET&A  pro¬ 
gram.  Additionally,  through  their  Computer  Security  Resource  Center  web  site,  NIST 
provides  a  mechanism  to  promulgate  timely  information  to  the  information  security 
community.  The  web  site  address  for  the  ET&A  information  can  be  found  at 

http://csrc.nist.gov/ATE. 

D.3.9  National  Security  Systems  education  &  training  requirements 

As  the  National  Manager  for  NSS,  NSA  is  responsible  for  development  of  standards  for 
education  and  training  for  the  national  security  community.  It  has  issued  two  policy  di¬ 
rectives  and  four  instructions  (during  the  period  19xx-200x)  containing  national  training 
standards  for  individuals  with  responsibilities  in  information  security.  These  documents 
reflect  NSA’s  understanding  that  individuals  with  differing  responsibilities  in  information 
security  need  different  levels  and  types  of  training.  They  provide  a  detailed  roadmap  for 
Federal  agencies  with  national  security  systems  to  incorporate  into  their  own  programs. 
The  specific  documents  are: 

D.3.9.1  NSTISSD  No.  500,  “Information  Systems  Security  (INFOSEC)  Education, 
Training  and  Awareness”,  25  February  1993. 

D.3.9.2  NSTISSD  No.  501,  “National  Training  Program  for  Information  Systems  Se¬ 
curity  (INFOSEC)  Professionals”,  16  November  1992. 


132  5  CFR  Part  930,  subpart  C,  “Information  Security  Responsibilities  for  Employees  who  Manage  or  Use 
Federal  Information  Systems”,  14  June  2004. 

133  NIST  SP  800-16,  Information  Technology  Security  Training  Requirements :  A  Role-  and  Performance- 
Based  Model,  April  1998  and  NIST  SP  800-50,  Building  an  Information  Technology  Security  Awareness 
and  Training  Program,  October  2003. 
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D.3.9.3  NSTISSI  No.  4011,  “National  Training  Standards  for  Information  Systems 
Security  (INFOSEC)  Professionals”,  20  June  1994. 

D.3.9.4  CNSSI  No.  4012,  “National  Information  Assurance  Training  Standards  for 
Senior  System  Managers”,  June  2004. 

D.3.9.5  CNSSI  No.  4013,  “National  Information  Assurance  Training  Standard  for  Sys¬ 
tems  Administrators”,  March  2004. 

D.3.9.6  CNSSI  No.  4014,  “National  Assurance  Training  Standard  for  Information  Sys¬ 
tems  Security  Officers”,  April  2004. 

D.3.9.7  NSTISSI  No.  4015,  “National  Training  Standards  for  System  Certifiers”,  De¬ 
cember  2000. 

D.3.10  DoD  Education  &  Training  Requirements 

DoD  has  a  detailed  education,  training  and  awareness  programs  for  its  personnel  for  a 
variety  of  areas  and  a  rigorous  program  of  how  to  do  education,  training  &  awareness 
(ET&A),  particularly  for  its  unifonned  personnel  since  their  ET&A  is  tied  to  career  fields 
at  both  the  enlisted  and  officer  level.  There  are  three  primary  documents  addressing  in¬ 
formation  security/awareness  training:  (1)  DoDD  8500.1  which  contains  the  primary  di¬ 
rection  for  I A  training  for  all  and  for  specific  responsibilities  for  IA134;  (2)  DoDI  8500.2 
that  provides  some  additional  details  on  who  should  be  trained  in  IA  ;  and  (3)  DODD 
8570.1  that  provides  extensive  detail  on  ET&A  for  the  IA  workforce,  including  certifica- 
tion  requirements  for  individuals  with  certain  designated  responsibilities  for  IA  ~  .  There 
is  a  companion  manual  for  the  last  document,  which  provides  guidance  and  procedures 
for  the  IA  workforce  ET&A  in  draft  that  is  awaiting  final  approval. 

D.3.11  Intelligence  Community  Education  &  Training  Requirements 

In  DCID  6/3,  the  DCI  is  required  to  establish  and  maintain  a  fonnal  information  security 
education,  awareness  and  training  program.  Also  included  were  the  agencies,  depart¬ 
ments  and  components  of  the  IC.  Specifically,  since  this  document  primarily  addresses 
the  Certification  and  Accreditation  process  (C&A)  for  allowing  systems  to  operate  and 
connect  in  the  IC,  the  education,  training  and  awareness  requirements  are  geared  towards 
individuals  who  are  involved  in  this  process.  DCID  6/3  spells  out  the  details  of  the  edu¬ 
cation,  training  and  awareness  requirements  and  included  that  for  individuals  with  spe¬ 
cific  responsibilities  for  the  C&A  process,  as  well  as  what  general  users’  training  should 
address.137 


134  DoDD  8500.1,  “Information  Assurance”,  24  October  2002,  para  4.22. 

135  DODI  8500.2,  “Information  Assurance  Implementation”,  6  Feb  2003,  para  5.7.7. 

136  DODD  8570.1,  “Information  Assurance  Training,  Certification,  and  Workforce  Management”,  15  Au¬ 
gust  2004. 

137  DCID  6/3,  “Protecting  Sensitive  Compartmented  Information  within  Information  Systems”,  Policy,  5 
June  1999,  para  B.3.C.;  Manual,  para  8.b.l. 
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D.4  Research 

This  section  examines  the  statutory  and  policy  requirements  for  the  NIAP  research.  The 
review  of  relevant  statutes  and  policies  for  the  NIAP  research  guidance  encompassed  all 
of  the  documents  discussed  in  the  body  of  this  report,  but  only  those  documents  specifi¬ 
cally  mentioning  research  are  discussed  here.  The  foundation  for  this  portion  of  the  re¬ 
port  is  the  relevant  portion  of  the  NIAP  memorandum  of  agreement  and  the  stated  and 
implicit  agreement  within  it  to  undertake  the  research  required  to  achieve  the  NIAP’s 
stated  objectives.  Specifically,  in  the  memorandum,  the  two  parties  agree  to  employ  the 
latest  techniques  to  develop  specification-based  tests,  methods,  and  tools  to  provide  ob¬ 
jective  measures  for  evaluating  quality  and  security.  They  also  commit  to  collaborative 
research  to  develop  the  test  methods.  The  research  required  to  develop  objective  meas¬ 
ures  has  been  lacking.  However,  this  is  not  the  only  NIAP-related  research  shortfall. 

Because  of  the  volatility  of  the  cybersecurity  arena,  due  to  new  attacks  being  invented 
each  day  as  well  as  the  deployment  of  new  information  technologies,  there  is  a  significant 
need  for  research  to  develop  the  tools  and  techniques  needed  to  continue  to  effectively 
execute  NIAP-related  activities.  However,  to  date,  very  little  of  the  necessary  NIAP- 
specific  research  infrastructure  has  been  assembled  and,  as  a  result,  the  NIAP  functions 
by  exploiting  research  developments  achieved  for  other  purposes.  Because  the  NIAP  is 
forced  to  re-purpose  research  results,  there  are  technological  aspects  of  the  NIAP  process 
that  remain  uninvestigated  even  though  these  technology  aspects  are  important  to  the 
NIAP  process. 

To  clearly  present  the  statutory  and  policy  guidance  as  it  relates  to  the  NIAP,  each  portion 
will  be  examined  in  turn.  The  following  table  (Table  21)  presents  the  statutory  and  policy 
provisions  regarding  research  that  are  NIAP  related. 
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Table  21.  Legislation  Regarding  NIAP-Related  Research 


DoD 

NIAP-related 

NIST 

NSF 

NIST 

Other  Federal 

Research  on  vulnerability 
assessments  and  techniques 
for  quantifying  risk 

CCA  1996 

Cyber  security  research, 
development  of  research 
centers,  research  grants, 
fellowships,  education 

ND.AA2001 

CSR&DA  2002 

CCA  1996 

HSA2002 

CSA  1987 

Determine  nature  and  extent 
of  computer  vulnerabilities, 
develop  cost-effective 
computer  security  techniques, 

HSA2002 

FISMA  2002 

3)  E-Gov  2002 

Review  private  sector 
information  security  policies, 
assess  standards  and 
guidelines,  and  apply  select 
critical  infrastructure  protection 
technologies  to  other  systems 

HSA  2002 

Assess  utility  of  tamper- 
re  s  i  sta  nt  s  e  cu  rity  s  oftwa  re 
and  other  security  tools  to 
protect  critical  C3I 

ND.AA  2004 

DARPA  and  NSA  coordinate 
research  efforts  for 
information  assurance 
research 

DoDD  8500.1 

D.4.1  Relevant  Statutory  Provisions 

Regarding  the  need  for  further  research  within  the  statutory  arena,  overall  there  appears 
to  be  an  implicit  assumption  that  there  will  be  adequate  research  conducted  to  support 
attainment  of  the  NIAP  goals.  While  some  statutes  contain  mandates  for  research,  most 
are  by  and  large  silent  on  the  subject  of  NIAP -relevant  research. 

D.4.2  Computer  Security  Act  1987  (CSA  1987) 

This  statute  calls  for  a  research  program  in  computer  security  and  for  research  to  protect 
computer  assets  from  attack.  None  of  the  research  called  for  in  the  act  is  directly  targeted 
at  supporting  the  NIAP,  but  some  results,  such  as  metrics  development,  that  would  derive 
from  the  research  that  is  suggested  could  serve  to  improve  the  NIAP  process. 

D.4.3  Clinger-Cohen  Act  1996  (CCA  1996) 

CCA  1996  authorizes  expenditures  for  cybersecurity  research  to  enhance  computer  secu¬ 
rity.  Within  the  act,  there  are  a  number  of  research  mandates  imposed,  most  of  which  are 
to  be  addressed  by  the  National  Science  Foundation.  However,  within  the  list  of  research 
areas  called  out  by  the  act,  there  is  only  one  that  is  applicable  to  the  NIAP.  That  provi¬ 
sion  calls  for  research  on  vulnerability  assessments  and  techniques  for  quantifying  risk. 
The  research  is  to  be  perfonned  under  the  aegis  of  the  National  Science  Foundation 
(NSF);  however,  since  NSF’s  mandate  is  to  pursue  fundamental/foundational  research, 
aligning  their  computer  security  research  goals  to  meet  even  this  need  of  the  NIAP  would 
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be  challenging  at  best.  Within  the  act,  other  appropriations  are  authorized  for  develop¬ 
ment  of  research  centers,  grants,  and  education  for  cybersecurity,  none  of  which  are  spe¬ 
cifically  targeted  at  the  NIAP  needs. 

D.4.4  Homeland  Security  Act  2002  (HSA  2002) 

This  statute  tasks  the  National  Institute  of  Science  and  Technology  to  perform  research 
related  to  computer  security,  to  determine  the  nature  and  extent  of  information  security 
vulnerabilities,  and  to  develop  techniques  for  providing  cost-effective  information  secu¬ 
rity.  The  act  also  tasks  NIST  to  review  policies  for  private  sector  information  security,  to 
determine  those  techniques  developed  to  protect  national  security  systems  that  can  be 
used  to  protect  other  systems  to  increase  their  information  security,  and  to  assess  stan¬ 
dards  and  guidelines.  Here  again,  the  research  tasking  does  not  directly  address  the  NIAP 
needs  and  the  NIAP  is  not  specifically  addressed  in  the  act. 

D.4.5  E-Government  Act  2002  (E-Gov  2002) 

This  Act  tasks  NIST  with  the  responsibility  to  conduct  research,  as  needed,  to  detennine 
the  nature  and  extent  of  information  security  vulnerabilities  and  techniques  for  providing 
cost-effective  infonnation  security.  However,  the  act  also  tasks  NIST  with  the  responsi¬ 
bility  for  developing  and  periodically  revising  performance  indicators  and  measures  for 
agency  information  security  policies  and  practices;  a  tasking  that  does  support  the  NIAP 
needs  to  some  degree. 

D.4.6  Federal  Information  Security  Management  Act  2002  (FISMA  2002) 

FISMA  2002  requires  NIST  to  conduct  research,  as  needed,  to  determine  the  nature  and 
extent  of  information  security  vulnerabilities  and  techniques  for  providing  cost-effective 
information  security. 

D.4.7  Cyber  Security  Research  and  Development  Act  2002  (CSR&DA  2002) 

This  statute  provides  $903  million  over  5  years  for  cybersecurity  research  and  education 
programs.  This  statute  directs  the  National  Science  Foundation  to  create  new  cybersecu¬ 
rity  research  centers,  program  grants,  and  fellowships.  The  statute  also  directs  NIST  to 
create  new  program  grants  for  partnerships  between  academia  and  industry. 

D.4.8  NDAA  2001 

DoD  was  required  by  this  Act  to  establish  an  Institute  for  Defense  Computer  Security  and 
Information  Protection.  The  Institute  would  conduct  research  relevant  to  foreseeable 
computer  and  network  security  requirements  and  information  assurance  requirements  of 
the  Department  of  Defense.  The  principal  foci  of  the  Institute  would  be  on  addressing 
research  areas  not  being  addressed  by  other  organizations  in  the  private  or  public  sector 
and  facilitating  the  exchange  of  information  regarding  cyberthreats,  technology,  tools, 
and  other  relevant  issues.  Grant,  education,  and  centers  of  excellence  programs  were  also 
established  and  funded. 

D.4.9  NDAA  2004 

This  Act  contains  no  direct  provisions  for  NIAP  research,  but  it  does  direct  the  Secretary 
of  Defense  to  assess  the  utility  of  tamper-resistant  security  software  and  other  innovative 
software  security  tools  in  protecting  critical  DoD  command,  control,  communications  and 
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intelligence  software  and  to  incorporate  such  protections  where  they  are  effective,  which 
by  implication  requires  a  study/survey  of  technologies.  The  results  of  this  study/survey 
would  be  applicable  to  the  NIAP. 

D.4.10  The  Health  Insurance  Portability  And  Accountability  Act  Of  1996 

This  Act  contains  no  provisions  for  information  security  research  or  information  assur¬ 
ance  research.  We  mention  this  act,  even  though  it  contains  no  research  provisions,  to 
highlight  the  fact  that  this  act  imposes  implicit  demands  for  major  advances  in  cybersecu¬ 
rity  but  does  not  discuss  these  advances,  provide  a  framework  for  them,  or  mandate  them. 
The  Gramm-Leach-Bliley  Act  also  contains  no  provisions  for  information  security  re¬ 
search  or  information  assurance  research,  but  is  mentioned  for  the  same  reason  as  the 
Health  Insurance  Portability  And  Accountability  Act  Of  1996. 

D.4.11  Relevant  Policy  Provisions 

D.4.11.1  National  Strategy  to  Secure  Cyberspace 

This  document  states  that  the  Federal  government  should  support  research  and  technol¬ 
ogy  development  to  enable  the  private  sector  to  better  secure  privately-owned  portions  of 
the  national  critical  infrastructure.  The  document  also  points  out  that  the  DHS  is  respon¬ 
sible  for  performing  and  funding  research  and  development  leading  to  new  scientific  un¬ 
derstanding  and  technologies  to  support  homeland  security,  and  by  implication  cyber¬ 
space.  The  document  also  recommends  that  the  Federal  government  prioritize  cybersecu¬ 
rity  research  and  development  agendas,  as  one  of  eight  major  actions  to  be  undertaken.  It 
is  pointed  out  that  “An  important  goal  of  cybersecurity  research  will  be  the  development 
of  highly  secure,  trustworthy,  and  resilient  computing  systems,”  but  specifics  concerning 
prioritization  or  process  to  achieve  this  objective  are  not  discussed.  The  emergence  of 
new  technologies  in  cyberspace,  such  as  optical  computing  and  nanotechnology,  are 
raised  as  factors  influencing  cybersecurity  in  the  future.  Several  research  priorities  are 
identified,  but  neither  the  NIAP  nor  any  of  its  supporting  technologies  are  among  them. 

D.4.11.2  HSPD  7 

This  directive  does  establish  some  research  policy  guidance.  It  requires  the  Department 
of  Commerce  to  work  with  private  sector,  research,  academic,  and  government  organiza¬ 
tions  to  improve  technology  for  cyber  systems  and  promote  other  critical  infrastructure 
efforts.  This  activity  includes  using  Commerce’s  authority  under  the  Defense  Production 
Act  to  assure  the  timely  availability  of  industrial  products,  materials,  and  services  to  meet 
homeland  security  requirements. 

D.4.11.3  DODD  8500.1 

requires  the  director  of  DARPA  to  coordinate  research  efforts  in  the  information  assur¬ 
ance  arena  with  the  director  of  the  NSA. 

D.4.12  Research  Implementation  Issues 

The  documents  referenced  in  the  next  section  discuss  implementation  issues  for  Re¬ 
search. 
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D.4.12.1  Status  of  the  Federal  Critical  Infrastructure  Protection  Activities 

This  2001  report  to  the  President  examines  the  needs  and  accomplishments  of  the  differ¬ 
ent  departments  and  agencies  of  the  Federal  government  as  regards  Critical  Infrastructure 
Protection  as  of  the  date  of  the  report.  The  report  points  out  the  need  for  research  across 
a  broad  front  that  addresses  a  wide  variety  of  cybersecurity  research  topics.  The  topics 
that  are  identified  are  department  and  agency  specific  as  well  as  overarching  research 
needs,  some  of  which  may  be  re-purposed  for  use  within  the  NIAP. 

D.4.12.2  Information  Security  and  Privacy  Advisory  Board 

An  ISPAB  report  in  2004  takes  note  of  the  need  for  additional  funding  in  the  area  of  cy¬ 
bersecurity  and  for  developing  standards  and  guidelines  that  serve  to  protect  the  cyber 
infrastructure.  The  Board  is  critical  of  progress  to  date  and  that  “Legislation  enacted  by 
Congress  in  recent  years  (e.g.,  FISMA  and  the  Cyber  Security  R&D  Act)  suggests  that 
the  Congress  recognizes  that  need  but  the  programs  authorized  in  those  Acts  have  not 
been  funded.”  Additionally,  they  note  that  “While  funding  for  the  program  in  real  terms 
has  grown  modestly,  it  has  not  kept  pace  with  the  growing  demand  for  cybersecurity 
guidelines  and  standards  as  a  result  of  the  government  and  the  nation’s  growing  reliance 
of  information  technology,  the  growth  and  diversity  of  the  technologies  on  which  we 
have  come  to  depend,  and  the  increased  threat.”  They  conclude  that  current  levels  of 

138 

funding  are  inadequate. 

D.4.12.3  General  Accountability  Office  (GAO)  reports 

D.4. 12.3.1  GAO  2003  High-Risk  Series  report:  Protecting  Information  Systems: 

Supporting  the  Federal  Government  and  the  Nation’s  Critical  Infrastructures 

This  report  notes  the  need  for  continued  research  in  the  field  of  information  assurance. 
The  report  notes  that  there  is  an  ongoing  need  for  research  and  that  funding  for  informa¬ 
tion  assurance  research  has  been  authorized  over  a  five-year  period  from  2003-2008  but 
delves  no  deeper  into  the  research  issues  associated  with  information  security  or  the 
NIAP. 

D.4. 12.3.2  GAO  report  GAO-04-321  -  Cybersecurity  for  Critical  Infrastructure 

Protection 

In  this  report,  GAO  found  that  “Despite  the  availability  of  current  cybersecurity  tech¬ 
nologies,  there  is  a  demonstrated  need  for  new  technologies.  Long-term  efforts  are 
needed,  such  as  the  development  of  standards,  research  into  cybersecurity  vulnerabilities 
and  technological  solutions,  and  the  transition  of  research  results  into  commercially 
available  products.”  In  the  report,  the  GAO  also  notes  that  “several  standards  exist  for 
cybersecurity  technology  in  the  areas  of  protocol  security,  product-level  security,  and  op¬ 
erational  guidelines;  there  is  still  a  need  to  develop  standards  that  could  help  guide  the 
use  of  cybersecurity  technologies  and  processes.”  Furthermore,  the  report  states  that 
there  is  a  need  for  research  to  address  future  security  needs,  “possible  long-tenn  research 
areas  include  tools  for  ensuring  privacy,  embedding  fault-tolerance  in  systems,  self- 


13S  ISPAB,  “NIST  Computer  Security  Division:  The  Case  for  Adequate  Funding”,  June  2004.  document 
available  at  http://csrc.nist.gov/ispab/bd-recommendations/ISPAB-ReportAdequateFundingNIST-CSD.pdf. 
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managing  and  self-healing  systems,  and  re-architecting  the  Internet”  as  well  as  research 
into  security  issues  related  to  control  of  critical  infrastructure.  The  report  reprises  the 
discussion  here  of  several  of  the  major  statutes  affecting  infonnation  security,  and  uses 
these  statutes  as  a  jumping-off  point  to  call  for  additional  research  into  security  technolo¬ 
gies  affected  by  the  statutes.  The  report  identifies  several  technological  areas  in  need  of 
further  research  efforts;  these  areas  include  1)  composing  secure  systems  from  insecure 
components,  2)  security  for  network  embedded  systems,  3)  security  metrics  and  evalua¬ 
tion,  4)  the  socioeconomic  impact  of  security,  5)  vulnerability  identification  and  analysis, 
and  6)  wireless  security.  Note  that,  while  not  specifically  mentioned,  the  need  for  further 
work  on  metrics  and  evaluation  is  relevant  to  the  NIAP  and  indicates  that  it  is  believed 
that  we  lack  adequate  means  for  assessing  the  security  properties  of  an  application. 

D.4.12.4  Department  of  Defense  Reports 

D.4.12.4.1  1996  Report  of  the  Defense  Science  Board  on  Information  Warfare 

Defense139 

This  report  states  that  there  is  a  need  for  an  ambitious  research  program  for  information 
warfare  defense.  However,  even  though  they  do  not  specifically  address  the  needs  of  the 
NIAP  directly,  they  do  call  for  a  vigorous  research  program  in  all  areas  of  information 
warfare  defense. 

D.4.12.4.2  2001  Report  of  the  Defense  Science  Board  on  Defensive  Information 

Operations140 

This  report  also  notes  the  need  for  a  vigorous  research  program  for  information  warfare 
defense,  especially  regarding  the  global  infonnation  grid  (GIG).  However,  the  research 
fields  that  they  identify  do  not  address  the  research  needs  of  the  NIAP.  For  example,  they 
note  the  need  for  research  in  scalable  global  computing,  mobile  code  security,  fault  toler¬ 
ance,  and  malicious  code  detection  but  do  not  call  for  research  to  improve  the  CCEVS  or 
the  NIAP  or  for  research  in  technologies  directly  related  to  them. 

D.4.12.4.3  1999  Report  of  the  Defense  Science  Board  on  Globalization  and  Secu¬ 

rity141 

This  report  notes  the  need  for  research  in  the  area  of  what  they  call  “pre-operational  in¬ 
tegrity”  of  software  systems,  but  they  propose  to  achieve  this  goal  by  the  use  of  red- 
teaming,  use  of  hackers,  and  research  into  smart  testing  while  also  calling  for  the  use  of 
the  NIAP.  They  do  not  call  for  additional  research  that  would  improve  the  capability  of 
the  NIAP  to  insure  pre-operational  integrity. 


139  OUSD(AT&L),  Report  of  the  Defense  Science  Board  Task  Force  on  Information  Warfare  Defense,  No¬ 
vember  1996,  Washington,  D.C.  document  available  at  http://www.acq.osd.mil/dsb/reports/iwd.pdf. 

140  OUSD(AT&L),  Protecting  the  Homeland:  Report  of  the  Defense  Science  Board  Task  Force  on  Defen¬ 
sive  Information  Operations,  2000  Summer  Study,  Volume  II,  March  2001,  Washington,  D.C.  Document 
available  at  http://www.acq.osd.mil/dsb/reports/dio.pdf. 

141  OUSD(AT&L),  Final  Report  of  the  Defense  Science  Board  Task  Force  on  Globalization  and  Security, 

December  1999,  Washington,  D.C.  Document  available  at 

http://www.acq.osd.mil/dsb/reports/globalization.pdf 
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D.4.12.4.4  Information  Assurance:  Legal,  Regulatory,  Policy,  and  Organizational 
Considerations 142 

This  1999  report  by  the  Joint  Staff  provides  an  overview  of  previous  reports  and  activities 
as  well  as  missions  and  roles  as  they  relate  to  information  assurance.  The  document  does 
discuss  the  need  for  cybersecurity  research  throughout  and  identifies  several  arenas 
where  research  is  needed,  but  provides  no  framework,  schedule,  or  prioritization  for  re¬ 
search. 

The  following  table  shows  the  relationships  of  the  reports  to  the  various  issues  described. 


Table  22.  Policy  and  Reports  Regarding  NIAP -Related  Research 


DoD 

Other  Federal 

NIAP 

Need  for  additional  funding  and 
research  in  cyberdefense 

1)  Information  Security  and 
Privacy  Advisory  Board 

2;  Status  ofthe  Federal  Critical 
Infrastructure  Protection 

Activities 

3)  Protecting  Information 
Systems:  Supporting  the 

Federal  Government  and 
Nation’s  Critical  Infrastructure 
(GAO.  2003) 

4)  Cyber  Security  for  Critical 
Infrastructure  Protection  (GAO. 
2004) 

Need  for  a  vigorous  research 
effort  in  all  areas  of  information 
warfare  defense 

Information  Assurance:  Legal, 
regulatory.  Policy,  and 
Organizational  Considerations 

1996  Report  ofthe  Defense 
Science  Board  on  Information 
Warfare  Defense 

Need  for  vigorous  research 
program  outside  of  NIAP  and 
using  non-NlAP  techniques 

2000  Report  of  the  Defense 
Science  Board  on  Information 
Warfare  Operations 

Research  to  insure  pre- 
operational  integrity  by  using 

NIAP  plus  other  techniques 

1999  Report  of  the  Defense 
Science  Board  on  Globalization 
and  Security 

Research  that  can  be  re¬ 
purposed  to  address  NIAP 
needs 

1)  Status  ofthe  Federal 

Critical  Infrastructure 

Protection  Activities _ 

2)  Cyber  Security  for  Critical 
Infrastructure  Protection 
(GAO.  2004) 

Research  to  enable  protection  of 
private-sector  owned  portions  of 
the  critical  infrastructure 

National  Strategy  to  Secure 
Cyberspace 

Requires  Department  of 
Commerce  to  improve  cyber 
s  e  cu  rity  te  ch  n  o  1  o  gy 

HSPD  7  (2003) 

D.5  Acquisition 

This  review  is  not  intended  to  be  an  exhaustive  review  of  acquisition  policy,  but  a  sample 
of  the  primary  policies  impacting  acquisition  of  IT  security  products/services. 

D.5.1  CSA 1987 

In  this  statute,  the  Administrator  of  GSA  was  charged,  under  the  authority  given  to  him 
by  the  Brooks  Act143,  with  developing  acquisition  guidance  for  Federal  government  IT 


14  Joint  Staff  J-6,  Information  Assurance:  Legal,  Regulatory,  Policy  and  Organizational  Considerations, 4th 
edition,  1999.  Washington,  D.C. 

143  40  USC  759(d)  Federal  Property  and  Administrative  Services  Act  of  1949. 
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(Federal  information  resources  management)  and  with  ensuring  that  this  guidance  was 
consistent  with  the  standards  for  computer  security  promulgated  by  NIST  and  made  man¬ 
datory  by  the  Secretary  of  Commerce. 

D.5.2  OMB  Circular  A-130 

This  document  contains  both  general  IT  acquisition  policy,  and  policy  specific  for  IT  se¬ 
curity.  For  the  general  IT  acquisition  policy,  OMB  directs  agencies  to  use  competition 
and  to  maximize  their  return  on  investment,  as  well  as  structuring  major  IT  systems  into 
segments  to  reduce  risk,  promote  flexibility  and  interoperability,  as  well  as  other  factors. 

For  policy  specific  to  IT  security,  and  consistent  with  and  implementing  the  requirements 
detailed  in  the  CSA,  OMB  directed  Commerce  to  issue  FIPS  and  guidelines  for  acquisi¬ 
tion,  and  GSA  to  provide  guidance  to  Federal  agencies  to  address  security  issues  when 
acquiring  IT.  The  direction  to  GSA  included  development  of  broad  contract  vehicles  to 
obtain  computer  security  products  and  services,  as  well  as  provide  cost-effective  security 
services  to  Federal  agencies.  GSA  implemented  these  requirements  by  issuing  the  Fed¬ 
eral  Information  Resources  Management  Regulation  (FIRMR)  144  and  the  Federal  Acqui¬ 
sition  Regulations  (FAR)145.  These  documents  provided  little  in  the  way  of  additional 
guidance  other  than  stating  that  Federal  agencies  must  comply  with  OMB  Circular  A-130 
in  addressing  security  considerations  in  the  procurement  of  IT. 

D.5.3  CCA  1996 

As  described  previously,  this  act  repealed  the  section  of  the  Federal  Property  and  Admini¬ 
stration  Services  Act  of  1949  (40  U.S.C.  750,  section  111),  resulting  in  rescission  of  the 
FIRMR  mentioned  above.  It  focused  instead  on  the  authority  of  the  Director,  OMB,  to 
oversee  Federal  agency  budget  and  programs  in  IT  capital  planning  and  investments.  This 
oversight  intended  to  use  perfonnance  and  results-based  management  to  ensure  appropri¬ 
ate  evaluations  of  Federal  agencies’  programs,  e.g.  OMB  to  ensure  that  the  infonnation 
security  policies,  procedures  and  practices  are  adequate.  146  It  also  provided  policy  direc¬ 
tion  for  the  process  of  acquisition  of  IT,  and  designated  the  Federal  Acquisition  Regula¬ 
tory  Council  as  the  responsible  agency  to  assure  that  the  process  for  acquisition  of  infor¬ 
mation  technology  was  simple,  clear  and  understandable. 

D.5.4  EO  13011  Federal  Information  Technology  1996 

This  EO  implements  the  requirements  for  IT  contained  in  CCA,  along  with  PRA 
1995147  and  GPRA  1993148.  It  reiterated  the  responsibilities  assigned  to  Federal 
agency  heads  for  complying  with  acquisition  guidance  contained  in  the  FAR  and  that  to 
be  issued  by  OMB  regarding  infonnation  systems  investments.  Although  CCA  repealed 


144  41  CFR  Part  201 

145  FAR  Part  30  Acquisition  of  Information  Technology 

146  CCA,  Section  5113,  “Performance  Based  and  Results-Based  Management”.  This  entire  section  empha¬ 
sizes  this  point. 

147  Public  Law  104-13,  “Paperwork  Reduction  Act” 

148  Public  Law  103-62,  “Government  Performance  Results  Act” 
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the  statute  underlying  GSA’s  authority  for  centralized  acquisition  of  IT  for  the  Federal 
government,  the  EO  directed  GSA  to  recommend  methods  and  strategies  for  acquisition 
of  IT.  No  specific  mention  of  computer  security  or  acquisition  of  security  products  or 
services  was  made  in  this  document. 

D.5.5  Federal  Acquisition  Regulation  (FAR)  subpart  239.71 149 

This  section  of  the  FAR,  titled  “Security  and  Privacy  for  Computer  Systems,”  was  set 
aside  to  provide  direction  to  Federal  agencies  on  IT  acquisitions  and  the  inclusion  of  in¬ 
formation  assurance  requirements.  Although  the  FAR  applies  to  all  Federal  departments 
and  agencies,  the  only  direction  it  currently  contains  is  very  general,  mentioning  CCA 
and  FIPS  standards  among  other  documents,  with  the  remainder  of  the  policy  focused  at 
DoD  and  national  security  systems  (described  in  a  little  more  detail  below  under  DoD 
policies).  There  are  a  number  of  FIPS  standards  applicable  to  all  Federal  agencies  (with 
the  exception  of  national  security  systems),  but  none  specifically  address  acquisition  of 
IA/IA-enabled  IT  for  Federal  agencies.  NIST  has  issued  guidance  for  acquisition  of  IT 
security  products  in  two  special  publications.  150  The  guidance  provided  in  these  docu¬ 
ments  is  not  mandatory  or  binding  on  the  part  of  Federal  agencies,  but  does  provide  a  ba¬ 
sis  for  agencies  to  determine  what  they  should  acquire. 

D.5.6  NTISSP  11  National  Policy  Governing  the  Acquisition  of  Information  As¬ 
surance  (IA)  and  IA-Enabled  Information  Technology  (IT)  Products 

This  policy,  applicable  only  to  the  national  security  community,  became  effective  in 
January  2001  and  established  a  policy  that  by  July  2002,  all  IA  or  IA-enabled  products 
purchased  by  this  community  must  be  evaluated  and  validated  by  one  of  three  methods: 
(1)  the  International  Common  Criteria  for  Information  Security  Technology  Evaluation 
Mutual  Recognition  Arrangement  (Common  Criteria);  (2)  NIAP;  or  (3)  NIST  FIPS  vali¬ 
dation  program.  It  provides  for  exemptions  and  deferred  compliance.  The  policy  also 
states  that  since  these  products  are  normally  part  of  a  system,  along  with  other  products,  a 
solution  security  analysis  be  conducted  along  with  the  certification  and  accreditation 
process,  in  addition  to  the  evaluation  and  validation  of  specific  IA/IA-enabled  products. 
There  are  no  requirements  that  provide  for  integration  of  the  product  evaluation  data  with 
C&A  of  systems  using  those  products. 

D.5.7  National  Defense  Authorization  Act  (NDAA)  FY  2001 151 

Section  8 1 1  of  this  Act  provided  direction  to  the  DoD  CIO  on  mission  critical  and  mis¬ 
sion  essential  IT  systems,  specifically,  directing  the  revision  of  DoDD  5000.1  to  establish 
minimum  planning  requirements  for  the  acquisition  of  IT  systems.  This  change  man¬ 
dated  these  systems  must  be  registered  with  the  CIO,  that  there  be  an  appropriate  infor- 


149  48  CFR  Ch.  2  subpart  239.71 

150  NIST  SP  800-23,  Guidelines  to  Federal  Organizations  on  Security  Assurance  and  Acquisition/Use  of 
Tested/Evaluated  Products,  U.S  Department  of  Commerce,  August  2000,  and  NIST  SP-800-36,  Guide  to 
Selecting  Information  Technology’  Security’  Products,  U.S.  Department  of  Commerce,  October  2003. 

151  Public  Law  106-398,  National  Defense  Authorization  Act,  FY  2001.  Title  VIII,  Subtitle  B,  "Information 
Technology",  Section  811,  "Acquisition  and  Management  of  Information  Technology",  30  October  2000. 
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mation  assurance  strategy,  and  that  certain  milestone  requirements  be  adhered  to  for  ma¬ 
jor  automated  infonnation  systems  (MAIS).  Additionally,  DoD  was  required  to  track 
purchases  of  IT  products  or  services,  in  excess  of  the  simplified  acquisition  threshold, 
regardless  of  the  method  of  purchase  to  ensure  compliance  with  sections  5122  and  5123 
of  CCA. 


D.5.8  E-Gov  Act  2002 

Among  other  things,  this  Act  created  the  Office  of  Electronic  Government  within  OMB 
to  oversee  Federal  government  efforts  in  accomplishing  e-gov  requirements,  including 
ensuring  that  products/services  were  acquired  appropriately  through  capital  planning  and 
investment  control  for  information  technology  as  required  under  CCA  .  It  formally 
chartered  the  CIO  Council,  previously  created  by  EO  13011,  and  designated  this  group  as 
the  principal  interagency  forum  for  improving  agency  practices  related  to  the  design,  ac¬ 
quisition,  development,  modernization,  use,  operation,  sharing  and  performance  of  Fed- 

1  r-S 

eral  Government  information  resources.  The  Act  also  authorized  heads  of  Federal 
agencies  to  enter  into  “share-in-savings  contracts”  for  IT  and  allowed  the  agencies  to  re¬ 
tain  savings  realized  by  these  contracts  (with  some  stipulations).  154  Finally,  this  act  au¬ 
thorized  state  and  local  government  to  acquire  IT  through  federal  supply  schedules.  155 

D.5.9  FISMA  2002 

FISMA  has  one  concern  with  regard  to  the  acquisition  of  IT  products  i.e.  a  statement  con¬ 
cerning  the  development  of  standards  by  NIST.  Federal  agencies  are  required  to  follow 
the  standards  and  guidelines  developed  by  NIST: 

(1)  to  the  maximum  extent  practicable,  ensure  that  standards  and  guidelines  do  not 
require  the  use  of  procurement  of  specific  products,  including  any  specific  hard¬ 
ware  or  software; 

(2)  to  the  maximum  extent  practicable,  ensure  that  such  standards  and  guidelines  pro¬ 
vide  for  sufficient  flexibility  to  pennit  alternative  solutions  to  provide  equivalent 
levels  of  protection  for  identified  infonnation  security  risks;  and 

(3)  to  the  maximum  extent  practicable,  use  flexible,  perfonnance-based  standards  and 
guidelines  that  pennit  the  use  of  off-the-shelf  commercially  developed  informa¬ 
tion  security  products.156 

D.5.10  National  Defense  Authorization  Act  (NDAA)  for  Fiscal  Year  2003 

The  NDAA  for  Fiscal  Year  2003  included  the  following  statement  of  policy  regarding 
DoD  acquisitions: 


152  PL  107-347,  Section  3602,  (e)(1). 

153  Ibid,  Section  3603 

154  PL  107-347,  section  210. 

155  Ibid,  section  211 

156  Ibid,  section  303,  para  (c)(5),  (c)(6),  and  (c)(7). 
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SEC.  352.  POLICY  REGARDING  ACQUISITION  OF  INFORMATION 
ASSURANCE  AND  INFORMATION  ASSURANCE-ENABLED 
INFORMATION  TECHNOLOGY  PRODUCTS. 

(a)  Establishment  of  Policy. — The  Secretary  of  Defense  shall  establish  a  policy  to 
limit  the  acquisition  of  information  assurance  and  information  assurance-enabled 
information  technology  products  to  those  products  that  have  been  evaluated  and 
validated  in  accordance  with  appropriate  criteria,  schemes,  or  programs. 

(b)  Waiver. — As  part  of  the  policy,  the  Secretary  of  Defense  shall  authorize 
specified  officials  of  the  Department  of  Defense  to  waive  the  limitations  of  the 
policy  upon  a  determination  in  writing  that  application  of  the  limitations  to  the 
acquisition  of  a  particular  information  assurance  or  information  assurance  en¬ 
abled  information  technology  product  would  not  be  in  the  national  security  inter¬ 
est  of  the  United  States. 

(c)  Implementation. — The  Secretary  of  Defense  shall  ensure  that  the  policy  is 
uniformly  implemented  throughout  the  Department  of  Defense.  157 

D.5.11  DoD  policies  regarding  acquisition  and  information  assurance 

The  DoD  has  been  the  most  active  of  the  Federal  departments/agencies  in  recognizing  the 
need  for  computer  security  in  its  systems.  Historically,  the  national  security  systems  (for 
the  most  part-  the  classified  systems  within  the  Department)  were  of  the  most  concern 
and  warranted  close  scrutiny  of  the  IT  products  and  systems  used.  There  have  been  a 
number  of  mechanisms  used  by  the  DoD  to  ensure  the  security  of  products  and  systems, 
but  for  the  purposes  of  this  report,  only  those  currently  in  existence  are  discussed.  The 
policy  for  DoD  systems  is  comprehensive  and  provides  significant  detail  in  the  acquisi¬ 
tion  and  management  of  DoD  IT  systems.  The  following  discussion  covers  the  highlights 
of  the  major  policies.  Additionally,  each  of  the  major  DoD  departments/agencies  (includ¬ 
ing  the  military  services)  has  issued  implementing  direction  specific  to  their  organization. 
The  details  of  that  level  of  policy  and  whether  that  direction  is  consistent  with  DoD  pol¬ 
icy,  sufficient  to  ensure  implementation  and  actually  implemented  as  intended,  will  not  be 
addressed  in  this  report,  but  may  be  appropriate  for  another  study. 

D.5.11. 1  DFAR  (Defense  Federal  Acquisition  Regulations)  June  2004 158 

This  regulation  codified  the  policy  for  national  security  DoD  IT  systems  that  mandated 
the  provision  of  information  assurance  for  IT,  including  the  requirement  for  IA/IA- 
enabled  products  to  be  evaluated  and  validated  by  the  NIAP. 

D.5.11.2  DODD  5000.1  The  Defense  Acquisition  System159 

This  directive  instructed  DoD  acquisition  managers  to  address  infonnation  assurance  for 
DoD  systems  including  weapons  systems,  C4ISR  systems  and  other  IT  systems.  It  re¬ 
ferred  to  DoDD  8500.1  for  more  detailed  information,  (see  below) 


157  Bob  Stump  National  Defense  Authorization  Act  for  Fiscal  Year  2003,  P.L.  107-3 14 
18  48  CFR  Parts  239  and  252 

159  Department  of  Defense  Directive  5000.1,  “The  Defense  Acquisition  System”,  12  May  2003. 


D-38 


D.5.11.3  DoDI  5000.2  Operation  of  the  Defense  Acquisition  System160 

This  instruction  stated  that  contracts  for  awards  of  mission-critical  or  mission-essential  IT 
systems  cannot  be  awarded  until  the  DoD  CIO  has  determined  that  the  system  has  an  ap¬ 
propriate  information  assurance  strategy.  This  was  done  as  part  of  the  structured  acquisi¬ 
tion  process  for  DoD  IT  systems. 

D.5.11.4  DoDD  8000.1  Management  of  DoD  Information  Resources  and  Informa¬ 
tion  Technology161 

This  directive  provided  the  definitive  policy  to  DoD  departments/agencies  on  how  they 
are  to  manage  their  IT  systems.  It  included  direction  to  include  information  assurance 
requirements  during  functional  process  reengineering,  outsourcing  and  information  sys¬ 
tems  design  prior  to  acquisition  of  new  or  redesign  of  existing  IT  systems. 

D.5.11.5  DoDD  8500.1  Information  Assurance162 

This  directive  provided  the  definitive  policy  to  DoD  departments/agencies  on  how  infor¬ 
mation  assurance  will  be  addressed  in  DoD  IT  systems.  It  reiterated  that  IA  requirements 
will  be  addressed  in  the  acquisition  of  IT  systems,  including  the  requirement  for  all 
IA/IA-enabled  systems  to  comply  with  NSTISSP  11  requirements  for  evaluation  and 
validation.  Although  NSTISSP  1 1  applies  to  national  security  systems,  this  directive  ex¬ 
tended  the  requirement  to  all  DoD  IT  systems  whether  they  are  national  security  systems 
or  not. 

D.5.11.6  DoDI  8500.2  Information  Assurance  Implementation163 

This  instruction  provided  implementation  direction  for  the  policies  laid  out  in  the  com¬ 
panion  directive  mentioned  previously.  It  provided  details  on  how  NSTISSP  11  would  be 
implemented  by  the  DoD  components  for  IA/IA-enabled  systems  and  the  use  of  protec¬ 
tion  profiles  and  security  targets  by  vendors  in  the  acquisition  process. 

D.6  Analysis  of  policies  by  community 

Using  the  communities  of  interest  described  in  the  previous  chapter,  the  policies  were 
analyzed  to  determine  what  policies  applied  to  what  communities  to  get  an  understanding 
of  the  complexities  of  the  policy  relationships.  The  numeric  results  of  this  analysis  are 
presented  at  Table  23.  What  the  table  does  not  show  is  the  potential  overlaps  in  commu¬ 
nities  of  interest  where  policies  intended  for  different  communities  are  applicable.  An 
example  of  this  overlap  would  be  DoD,  which  exists  as  a  Federal  department,  a  member 
(and  the  executive  agent)  for  the  National  Security  Community,  and  has  elements  that  are 
also  members  of  the  Intelligence  Community.  Since  the  focus  of  this  study  is  on  the  Fed¬ 
eral  government,  the  communities  of  interest  of  state,  local  and  tribal,  as  well  as  the  pri- 


160  Department  of  Defense  Instruction  5000.2,  “Operation  of  the  Defense  Acquisition  System”,  12  May 
2003 

161  Department  of  Defense  Directive  8000.1,  “Management  of  DoD  Information  and  Information  Technol¬ 
ogy”,  with  Change  1,  20  March  2002. 

162  Department  of  Defense  Directive  8500.1,  “Information  Assurance”,  24  October  2002. 

163  Department  of  Defense  Instruction  8500.1,  “Information  Assurance  Implementation”,  6  February  2003. 
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vate  sector,  were  not  addressed.  Also  included  in  this  analysis,  were  policy  requirements 
specific  to  particular  activities  of  interest,  specifically  NIST,  NSA  and  the  NIAP. 


Table  23.  Policy  Requirements  by  Community  of  Interest 


Stakeholder  Source 

Number  and  Type  of  Policy  Requirements 

Federal  Community 

33  requirements  (IA,  Acquisition,  Certification  Accreditation,  Stan¬ 
dards/Guidelines,  CIP,  &  Reporting) 

National  Security 
Community 

19  unique  requirements  (IA,  Acquisition,  Certification  &  Accreditation, 
and  Reporting) 

DoD 

50  unique  requirements  (IA,  Acquisition,  CIP,  Trusted  Computer  Sys¬ 
tems,  Certification  &  Accreditation,  Protection  Profiles  &  Standards) 

Intelligence  Commu¬ 
nity 

2  unique  requirements  (IA,  Certification  &  Accreditation) 

NIST 

14  unique  requirements  (IA,  Standards/Guidelines) 

NSA 

13  unique  requirements  (Acquisition,  Trusted  Computer  Systems,  Evalu¬ 
ated  Products,  Protection  Profiles,  standards/guidelines) 

NIAP 

15  unique  requirements  (General) 

Table  24  shows  the  mix  of  requirements  that  may  be  applicable  to  NIAP  evaluated  prod¬ 
ucts. 
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Table  24.  NIAP  Requirements  Matrix 


NIAP  Requirements 


/&/ 

J>S/ 


Treaties  &  Other  International  Agreements 


CC-CEM  97/01 7  (1997) 


Strategi 


NS  for  Homeland  Security  (2002) 


NS  for  Physical  Protection  of  CI/KA  (2003) 


NS  to  Secure  Cyberspace  (2003) 


NSD  42  (1990) _ 

PDD63  (1998) 


HSPD7  (2003) 


FAR  Part  39 


5  CFR  Part  930  (2004)(lnfo  Sec  Resp  for 
Employees  who  Manage  or  Use  Federal  IS) 


15  CFR  Part  287  (2000)  (Guidance  on 
Conformance  Assessment  Activities) 


48  CFR  Parts  239  &252  (DFARS) _ 


OMB  CircA-119  (1998) 


OMB  CircA-130  (2000) 


OMB  M- 00-07 


OMB  M-01-08  (GISRA)  (2001) 


OMB  M-04-15  (HSPD7)  (2004) 
OMB  M- 04-25  (FISMA)  (2004) 


National  Security  Systems  Issuances 

NSTISSP  6  (1994)(C&A  for  NSS) _ 

NSTISSP  1 1  (2003)(Acquts ition  of  IA  &  IA- enabled 

Products) _ 

NSTISSD  501  (1992) (Training  for  Infosec  Prof) 

NSTISSP  502  (1993) (NSS  Security) _ 

NSTISS1 1000  (2000)(NIACAP) 

NSTISSI  401 1  (1994)(Training  Std  for  INFOSEC 

Professionals) _ 

CNSSI  401 2 (2004) (I A  Trng  StdforSSMs) 

CNSSI  4013(2004)(Trng  Std  for  SAs) _ 

CNSSI  40 1 4 (2004) (Trn g  Std  for  ISSOs) 

CNSSI  4015 (2000) (Trn g  Std  for  System  Certifiers) 
NSTISSAM INFOSEC/2-OO 


1C  Policy 

DCID6/3  (1999)(Policy  &Manual) 


DoD  Issuances 


DoDD  4630.5  (Interop  &  Supt  of  IT  &  NSS)(2004) 

DoDD  5000.1  (Defense  Acquis  ition)(2003) _ 

DoDD  5000.2  (Ops  of  Def  A  eg u is iti o n) (2003) 
DoDD  5160.54  (CAAP/CI P)(1 998) 

DoD  5200.40  (DITSCAP)  (1997) _ 

DoDD  5215.1  (1982)(CSEC) _ 

DoDD  8000. 1  (Mg ml  of  DoD  I RM  &  IT)(2002) 

DODD  8100.1  (GIG  Policy) (2002) _ 

DoD  8500.1  (IA)  (2002) _ 

DoDI  8500.2  (IA  lmpl)(20C6) _ 


D-41 


DODI 8551 .1  (Ports,  Protocol  Mgmt)(2004) 


DODD  8570. 1  (IA  Trng,  Certif  Workforce 
Mgmt)(2004) _ 


DoDI  8580.1  (IA  in  Def  Acq  Sys) 


CJCSI  6510.01  D  (2004)(IA  CND) 


FIPS  181  (1993) (Automated  Password  Generator) 


FIPS  199(2003) (Security  Categorization  of  Federal 
Info  &  Info  Systems) 


NISTSP  800-1 2(1 995) (Intro  to  Computer  S 


NIST  SP  800-1 4(1 996)  (Principles&Practices  fo 
Securing  IT  Systems) 


NISTSP  800-16  (1998)  ITS  Training  Rqmts 


NISTSP  800-1 8(1 998) (Developing  Security  Plans 
for  IT  Systems) 


NISTSP  800-21  (1999) (Implementing  Cryptography 
in  the  Federal  Government 


NIST  SP  800-23  (2000)(S ecurity  Assurance  & 
Acqui/Use  of  Tested/Evaluated  Products 


NISTSP  800-26(2001) (Security  Self-Assessment 
Guide  for  IT  Systems) _ 


NIST  SP  800-27 rev A(2004)(Engineering  Principles 
for  IT  Security) 


NISTSP  800-30(2002) (Risk  Management  Guide 
for  IT  Systems) _ 


NISTSP  800-31  (2001)  (IDS) 


NIST  SP  800-33(2001) (Underlying  Technical 
Models  for  IT  Security) 


NISTSP  800-34(2002) (Contingency  Planning 
Guide  for  IT  Systems 


NISTSP  800-36  (2003)(S electing  Information 
Security  Products) 


NIST  SP  800-37  (2002)(C&A  of  Federal  Systems) 


NISTSP 800-40(2002) (Security  Patches) 


NIST  SP  800-41  (2002) (Firewalls  &  FW  Policy) 


NIST  SP  800-42(2003) (Network  Security  Testing) 


NISTSP  800-44(2002)  (Securing  Public  Web 
Servers) 


NISTSP 800-45(2002) (Email  Security) 


NIST  SP  800-47 (2002) (Interconnecting  IT 
Systems) 


NISTSP  800-50  (2003)  IT  Security  A&T  Program) 


NISTSP  800-53  (2005)(Rec  Security  Controls) 


NISTSP  800-55(2003) (Security  Metrics  for  IT) 


NISTSP  800-59(2003) (ID  IS  as  a  NSS) 


NISTSP  800-60(2004) (Mapping  Types  of  Info  &IS 
to  Security  Categories) _ 


NIST  SP  800-63rev  1 .0. 1  (2004)(E lectronic 
Authentication  Guideline 


NISTSP  800-64  rev  1(2004)(S ecurity 
Considerations  in  IS  Dev  Life  Cycle) 


NISTSP  800-65(2005)  (Integrating  Security  into  the 
Capital  Planning  &  Investment  Control  Process) 


Nationalflnternational  Standards 


ISCVIEC65  (1996) 


IS0  17799  (2000)(lnformation  Security 
Management 


CC-CEVS  (2004)(l  SO/IEC  15408)v2.2 


CCArr  (2000) 
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D.7  Additional  Statutes  of  Interest 

In  general,  the  Federal  Government  does  not  directly  regulate  the  security  of  non¬ 
government  computer  systems.  However,  there  are  three  significant  exceptions  of  inter¬ 
est.  The  Federal  Government  does  require  certain  information  held  on  non-government 
systems  to  be  protected  against  unauthorized  access  and  disclosure,  primarily,  but  not  ex¬ 
clusively,  out  of  privacy  considerations.  To  date,  this  requirement  has  been  limited  to  cus¬ 
tomer  financial  information  (Gramm-Leach-Bliley  Act  of  1999  (GLB  Act)),  corporate 
financial  information  (Sarbanes-Oxley  Act  of  2002  (SOX  Act)),  and  medical  information 
(Health  Insurance  Portability  and  Accountability  Act  of  1996  (HIPAA)).  A  number  of 
regulatory  agencies  have  authority  for  developing,  promulgating,  and  enforcing  standards 
for  financial  information.  SOX  Act  requires  public  companies  to  certify  the  accuracy  of 
their  internal  financial  controls.  The  Securities  and  Exchange  Commission  (SEC)  has  au¬ 
thority  to  develop  standards  and  enforce  these  regulations.  The  Secretary  of  Health  and 
Human  Services  has  authority  under  HIPAA  to  develop  and  enforce  standards  for  medical 
information.  The  following  discussion  provides  more  detail  on  the  statutes  mentioned  and 
how  they  contribute  to  cybersecurity  within  the  private  sector. 

D.7.1  Gramm-Leach-Bliley  Act  of  1999  (GLB  Act)  164 

Title  V,  Subtitle  A  of  the  Gramm-Leach-Bliley  Act  addresses  privacy  of  consumer’s  fi¬ 
nancial  information  requiring  agencies  who  have  authority  in  the  financial  services  area 
(listed  below)  to  establish  appropriate  standards  for  financial  institutions  to:  (1)  insure  the 
security  and  confidentiality  of  customer  records  and  information,  (2)  protect  against  any 
anticipated  threats  of  hazard  to  the  security  or  integrity  of  such  records  and  (3)  to  protect 
against  unauthorized  access  to  or  use  of  these  records  that  could  result  in  harm  or  incon¬ 
venience  to  a  customer.  The  Act  directs  Federal  functional  regulators,  the  Treasury  Sec¬ 
retary,  and  the  Federal  Trade  Commission  (FTC),  in  consultation  with  State  insurance 
authorities,  to  prescribe  regulations  necessary  to  carry  out  the  purposes  of  the  Act,  mak¬ 
ing  every  effort  to  assure  such  regulations  are  consistent  and  comparable.165  The  FTC 
issued  its  final  rule  on  the  standards  for  safeguarding  customer  information  in  2002,  and 
provided  additional  guidance  to  businesses  to  which  these  rules  apply.166  The  rule  re¬ 
quired  all  financial  entities  subject  to  FTC’s  jurisdiction  to  develop,  implement  and  main¬ 
tain  comprehensive  information  security  programs,  appropriate  to  the  size  and  scope  of 
the  organization.  One  element  of  each  program  is  to  identify  risks  to  the  security,  confi¬ 
dentiality,  and  integrity  of  customer  information  in  information  systems,  including  net¬ 
work  and  software  design,  as  well  as  information  processing,  storage,  transmission  and 
disposal.  Other  agencies  with  regulatory  authorities  have  issued  similar  rules.  The  net 
effect  of  this  activity  is  that  almost  the  entire  financial  services  industry  is  required  to 


P.L.  106-102,  15  U.S.C.  sect  6801,  et  seq,  Title  V-Privacy,  Subtitle  A,  “Disclosure  of  Nonpublic  Per¬ 
sonal  Information”.  12  November  1999. 

165  Information  regarding  these  regulations  is  available  at  the  following  web  sites: 
http://www.keytlaw.com/Links/glbact.htm,  http://www.ftc.gov/privacy/glbact/index.html 

166  16  CFR  Part  314,  “Standards  for  Safeguarding  Customer  Information”,  23  May  2002. 
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provide  adequate  security  over  its  information  systems  to  ensure  compliance  with  this 
Act. 

D.7.2  Sarbanes-Oxley  Act  of  2002 167 

Section  404  of  the  Sarbanes-Oxley  Act  is  being  widely  interpreted  in  the  legal,  account¬ 
ing  and  information  assurance  communities  to  require  all  publicly  held  companies  to 
demonstrate  due  diligence  in  the  disclosure  of  financial  infonnation  and  implement  ap¬ 
propriate  internal  controls  and  procedures  to  communicate,  store  and  protect  that  data. 
Public  companies  are  also  required  to  protect  these  controls  from  internal  and  external 
threats  and  unauthorized  access,  including  those  that  could  occur  through  online  systems 
and  networks.  ISO  17799  is  the  recommended  standard  for  information  security  for 
public  companies  to  which  this  statute  applies.  The  Cyber  Security  Industry  Alliance 
(CSIA),  the  Information  Systems  Audit  and  Control  Association  (ISACA)  and  the  Com¬ 
puter  Security  Institute  (CSI),  among  others,  have  recognized  the  impact  of  this  statute 
and  authored  papers  or  conducted  studies  addressing  issues  in  this  area.  The  Public 
Company  Accounting  Oversight  Board  (PCAOB),  created  by  the  statute  to  provide  over¬ 
sight  for  the  implementation  of  this  statute,  has  issued  a  number  of  standards  and  rules, 
approved  by  the  SEC.  169  The  most  recent  CSI/FBI  Computer  Crime  and  Security  Sur¬ 
vey  added  a  question  concerning  the  effect  of  this  statute  on  information  security  activi¬ 
ties  that  CSI  will  continue  to  track  through  annual  surveys. 170 

D.7.3  Health  Insurance  Portability  and  Accountability  Act  of  1996 

HIPAA  imposes  stringent  requirements  on  health  care  providers  and  others  to  protect 
medical  information.  “Wrongful  disclosure  of  individually  identifiable  health  informa¬ 
tion”  is  a  criminal  offense.  HIPAA  requires  safeguards  on  the  part  of  anyone  who  main¬ 
tains  or  transmits  health  infonnation  to  ensure  the  integrity  and  confidentiality  of  the  in¬ 
formation  and  protect  against  any  reasonably  anticipated  “(i)  threats  or  hazards  to  the  se¬ 
curity  or  integrity  of  the  information;  and  (ii)  unauthorized  uses  or  disclosures  of  the  in- 
formation....”  HIPAA  does  not  distinguish  between  public  and  private  sector  organi¬ 
zations  in  imposing  these  requirements;  in  fact,  it  is  comprehensive  in  its  scope  Federal, 
state  and  local  entities  that  have  a  need  to  collect  health  information  on  individuals  are 
included,  as  well  as  private  sector  hospitals,  health  care  organizations  and  insurance 
companies.  All  are  required  to  provide  protection  for  individually  identifiable  health  in¬ 
formation.  The  Department  of  Health  and  Human  Services  (DHHS)  published  the  Pri- 


167  Public  Law  107-204,  116  stat.  745,  Section  404,  “  Management  Assessment  of  Internal  Controls”,  H.R. 
3763,  “Sarbanes-Oxley  Act  of  2002”,  24  July  2002. 

I6S  ISO  17799,  Information  technology’.  Code  of  practice  for  infonnation  security’  management,  First  Edi¬ 
tion,  2000-12-01. 

169  Web  site  address  is  www.pcaobus.org 

170  CSI.  “2004  CSI/FBI  Computer  Crime  and  Security  Survey”,  2004.  report  available  at 
http://i.cmpnet.com/gocsi/db_area/pdfs/fbi/FBI2004.pdf. 

171  Health  Insurance  Portability  and  Accountability  Act  of  1996,  P.L.  104-191,  §1 173(d)(2). 
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vacy  Rule  in  2002  "  and  the  Security  Rule  in  2003  ,  with  compliance  to  begin  in  April 

of  2003.  The  fact  that  a  Security  Rule  was  part  of  this  implementation  reinforces  the 
connection  between  Security  and  Privacy  and  provides  additional  incentive  for  organiza¬ 
tions  having  this  type  of  information  to  address  their  security  issues  in  a  more  general 
sense.  As  with  the  GLB  Act,  this  statute  covers  a  spectrum  of  organizations  in  the  af¬ 
fected  medical  health  services  industry,  thereby  mandating  information  security  for  an¬ 
other  large  group  of  public  and  private  sector  entities. 


172  DHHS,  “Standards  for  Privacy  of  Individually  Identifiable  Health  Information”,  45  CFR  Parts  160  and 
164,  14  August  2002. 

173  DHHS,  “Heath  Insurance  Reform:  Security  Standards”,  45  CFR  Parts  160,  162,  and  164,  20  Feb  2003. 
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Annex  E  NIAP  Historical  Data 


The  National  Information  Assurance  Partnership  (NIAP)  is  a  U.S.  Government  initiative 
originated  to  meet  the  security  testing  needs  of  both  information  technology  (IT)  con¬ 
sumers  and  producers.  The  NIAP  is  a  collaboration  between  the  National  Institute  of 
Standards  and  Technology  (NIST)  and  the  National  Security  Agency  (NSA)  in  fulfilling 
their  respective  responsibilities  under  PL  100-235  (Computer  Security  Act  of  1987).  The 
partnership  combines  the  extensive  IT  security  experience  of  both  agencies  to  promote 
the  development  of  technically  sound  security  requirements  for  IT  products  and  systems 
and  appropriate  measures  for  evaluating  those  products  and  systems. 

The  long-term  goal  of  the  NIAP  is  to  help  increase  the  level  of  trust  consumers  have  in 
their  information  systems  and  networks  through  the  use  of  cost-effective  security  testing, 
evaluation,  and  validation  programs.  In  meeting  this  goal,  the  NIAP  seeks  to: 

•  Promote  the  development  and  use  of  evaluated  IT  products  and  systems; 

•  Champion  the  development  and  use  of  national  and  international  standards  for  IT 
security; 

•  Foster  research  and  development  in  IT  security  requirements  definition,  test 
methods,  tools,  techniques,  and  assurance  metrics; 

•  Support  a  framework  for  international  recognition  and  acceptance  of  IT  security 
testing  and  evaluation  results;  and 

•  Facilitate  the  development  and  growth  of  a  commercial  security  testing  industry 
within  the  U.S. 

The  NIAP  Common  Criteria  Evaluation  and  Validation  Scheme  (CCEVS)  is  a  national 
program  for  the  evaluation  of  information  technology  products  for  conformance  to  the 
International  Common  Criteria  for  Information  Technology  Security  Evaluation 
(ISO/IEC  Standard  15408,  commonly  referred  to  as  the  Common  Criteria  (CC)).  The  CC 
represents  the  outcome  of  a  series  of  efforts  to  develop  criteria  for  evaluation  of  IT  security 
that  are  broadly  useful  within  the  international  community.  In  the  early  1980’s  the  Trusted 
Computer  System  Evaluation  Criteria  (TCSEC)  was  developed  in  the  United  States.  In  the 
following  decade,  various  countries  began  initiatives  to  develop  evaluation  criteria  that  built 
upon  the  concepts  of  the  TCSEC  but  were  more  flexible  and  adaptable  to  the  evolving  nature 
of  IT  in  general.  For  example,  The  European  Commission  published  the  information  Tech¬ 
nology  Security  Evaluation  Criteria  (ITSEC)  and  Canada  published  the  Canadian  Trusted 
Computer  Product  Evaluation  Criteria  (CTCPEC).  The  U.S.  government  published  the  Fed¬ 
eral  Criteria  for  Information  Technology  Security  (FC). 

In  1990  work  began  in  the  International  Organization  for  Standardization  (ISO)  to  develop  an 
international  standard  evaluation  criteria  for  general  use.  The  new  criteria  were  to  be  respon¬ 
sive  to  the  need  for  mutual  recognition  of  standardized  security  evaluation  results  in  a  global 
IT  market.  In  June  1993,  the  sponsoring  organizations  of  the  CTCPEC,  FC,  TCSEC  and 
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ITSEC  pooled  their  efforts  and  began  a  joint  activity  to  align  their  separate  criteria  into  a  sin¬ 
gle  set  of  IT  security  criteria  that  could  be  widely  used.  This  activity  was  named  the  CC  Pro¬ 
ject.  Its  purpose  was  to  resolve  the  conceptual  and  technical  differences  found  in  the  source 
criteria  and  to  deliver  the  results  to  ISO  as  a  contribution  to  the  international  standard  under 
development.  ISO/IEC  15408  was  approved  in  1999.  ISO/IEC  accepts  the  continued  use  of 
the  term  “Common  Criteria”  (CC)  within  the  document,  while  recognizing  that  its  official 
name  in  the  ISO  context  is  “Evaluation  Criteria  for  Infonnation  Technology  Security.” 

Table  25  below  is  a  chronology  of  computer  security  documents  and  events  that  preceded 
the  CCEVS  and  the  NIAP. 


Table  25.  Chronology  of  Computer  Security  Documents  and  Events 


Date 

Organization 

Document/Standard  /Project-Program/Legislation 

Short  Name 

10/67 

Defense  Science 

Board 

Task  Force  assembled  to  address  computer  security  safeguards  to  protect 
classified  information  in  remote-access,  resource-sharing  computer  systems. 
Report  later  issued  in  02/70 

02/70 

U.S.  DoD  (OSD) 

Security  Controls  for  Computer  Systems,  Report  for  Defense  Science  Board 
Task  Force  on  Computer  Security,  a  RAND  Report  that  made  a  number  of 
policy  and  technical  recommendations  to  reduce  threat  of  compromise  of 
classified  information  processed  on  remote-access  computer  systems 

10/72 

U.S.  DoD  (AFSC) 

Computer  Security  Technology  Study,  prepared  by  James  P.  Anderson  & 

Co.  for  the  Deputy  for  Command  and  Management  Systems,  HQ  Electronic 
Systems  Division  (AFSC).  Developed  solution  approaches  to  technical 
problems  associated  with  controlling  flow  of  infonnation  in  resource  and 
infonnation  sharing  computer  systems 

01/73 

U.S.  DoD 

DoD  5200. 28M,  ADP  Computer  Security  Manual—  Techniques  and  Proce¬ 
dures  for  Implementing,  Deactivating,  Testing  and  Evaluating  Secure  Re¬ 
source  Sharing  ADP  Systems  responded  to  DSB  recommendation  to  estab¬ 
lish  uniform  DoD  policy,  security  requirements,  administrative  controls  and 
technical  measures  to  protect  classified  infonnation  processed  by  DoD 
computer  systems. 

•  Established  COMPUSEC  principles 

•  Stated  that  security  testing  and  evaluation  procedures  would  be 
published  later  (Orange  Book) 

•  Laid  groundwork  for  Orange  Book 

1977 

U.S.  DoD 

DoD  Computer  Security  Initiative  created  under  auspices  of  the  Under 
Secretary  of  Defense  for  Research  and  Engineering  to  focus  DoD  efforts 
addressing  computer  security  issues. 

1977-1980 

National  Bureau 

of  Standards 

(NBS)  Later  NIST 

“Audit  and  Evaluation  of  Computer  Security,”  NBS  Special  Publication 
#500-19,  October  1977. 

“Processors,  Operating  Systems  and  Nearby  Peripherals:  A  Consensus 
Report,”  in  Audit  and  Evaluation  of  Computer  Security  II:  System  Vulner¬ 
abilities  and  Controls,  NBS  Special  Publication  #500-57,  MD78733,  April 

1980 

•  Began  effort  to  define  problems  and  solutions  for  building, 
evaluating  and  auditing  secure  computer  svstems 

•  Held  invitational  workshops  on  subject  of  audit,  and  evaluation 
of  computer  security 
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1979 

U.S.  DoD 

Proposed  Technical  Evaluation  Criteria  for  Trusted  Computer  Systems, 
published  in  support  of  the  DoD  Computer  Security  Initiative.  Authored  by 
MITRE  Corporation. 

•  Outgrowth  of  NBS  workshop  findings  and  recommendations 

06/79 

U.S.  DoD 

DoD  5200. 28M,  ADP  Computer  Security  Manual--  Techniques  and  Proce¬ 
dures  for  Implementing,  Deactivating,  Testing  and  Evaluating  Secure  Re¬ 
source  Sharing  ADP  Systems,  with  1 sl  Amendment 

01/81 

U.S.  DoD 

DoD  Computer  Security  Center  formed  to  expand  on  work  started  by  DoD 
Computer  Security  Initiative.  Goal:  To  encourage  widespread  availability 
of  trusted  computer  systems 

10/82 

U.S.  DoD 

DoD  5215.1,  Computer  Security  Evaluation  Center  established  DoD  Com¬ 
puter  Security  Evaluation  Center  (CSEC)  to  provide  policy  and  assign  re¬ 
sponsibilities  for  technical  evaluation  of  computer  system  and  network 
security. 

CSEC 

08/83 

U.S.  DoD 

CSC-STD-001-83,  Trusted  Computer  System  Evaluation  Criteria,  National 
Computer  Security  Center  (NCSC) 

•  Layered  approach  to  security  “strength”  rating 

•  Oriented  for  custom  software  running  on  mainframe  systems 

•  Difficult  to  apply  to  networks  and  databases,  interpretations  pro¬ 
vided. 

•  Evaluations  paid  for  by  Government 

TCSEC  or  Orange 

Book 

12/85 

U.S.  DoD 

DoD  Computer  Security  Center  renamed:  The  National  Computer  Security 
Center  (NCSC) 

•  Goal:  To  encourage  widespread  availability  of  trusted  computer 
products  for  use  on  systems  that  processed  classified  or  other 
sensitive  information 

•  Approach  to  produce  generic  requirements  that  could  be  used  by 
vendor  to  produce  trusted  products  and  would  serve  as  standard¬ 
ized  criteria  for  evaluating  the  trust  classes  of  those  products 

NCSC 

12/85 

U.S.  DoD 

DoD  5200.28-STD,  Trusted  Computer  System  Evaluation  Criteria,  National 
Computer  Security  Center  (NCSC) 

Superseded  CSC-ST-001-83 

•  Defined  seven  security  levels  for  trusted  hardware,  software  and 
data  components  of  a  system 

•  Goal:  To  provide  a  level  of  measurement  and  guidance  in  de¬ 
signing  secure  systems 

•  TCSEC  Standard  served  to:  1)  Provide  product  manufacturers 
with  a  standard  of  security  features  to  build  into  products;  2) 
Provide  DoD  components  with  metric  to  evaluate  how  much 
trust  could  be  placed  in  an  automated  information  system;  and  3) 
Provide  a  basis  for  specifying  security  requirements  in  acquisi¬ 
tion  specifications 

•  Under  the  Trusted  Product  Evaluation  Program  (TPEP),  vendors 
approached  NS  A  with  COTS  products  and  requested  evaluation 
Evaluators  under  TPEP  used  TSEC  and  its  interpretations  to  ac¬ 
cess  how  well  products  met  requirements  for  targeted  rating 

•  Designed  for  government  installations  not  corporate  networks 

•  Designed  for  standalone  systems 

•  Appendix  A:  Commercial  Product  Evaluation  Process 

TCSEC  or  Orange 

Book 
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12/85 

U.S.  OMB 

Office  of  Management  and  Budget,  Appendix  III  to  OMB  Circular  No.  A- 
130,  Security  of  Federal  Automated  Information  Resources, 

•  Required  that  federal  agencies  assure  each  system  appropriately 

uses  effective  security  products  and  techniques  consistent  with 
standards  and  guidance  from  NIST 

07/87 

U.S.  DoD 

NCSC-TG-005,  vl.0,  Trusted  Network  Interpretation  of  the  TCSEC,  Na¬ 
tional  Computer  Security  Center 

TNI,  Part  of  Rain¬ 
bow  Series 

Red  Book 

01/88 

U.S.  Congress 

Computer  Security  Act  of  1987,  Public  Law  100-235  (H.R.  145) 

•  Amended  the  *National  Bureau  of  Standards/NBS  Organic  Act 

of  1901,  P.L.  56-177  and  assigned  responsibility  to  NBS  for  de¬ 
veloping  standards  and  guidelines  for  federal  computer  systems, 
drawing  on  technical  advice  and  assistance  from  NS  A. 

*The  Omnibus  Trade  and  Competitiveness  Act  of  1988,  P.L.  100-418, 
changed  the  name  of  the  National  Bureau  of  Standards  (NBS)  to  the  Na¬ 
tional  Institute  of  Standards  and  Technology  (NIST) 

3/89 

NIST/NSA 

Memorandum  of  Understanding  Between  the  Director  of  the  National  Insti¬ 
tute  of  Standards  and  Technology  and  the  Director  of  the  National  Security 
Agency  Concerning  the  Implementation  of  Public  Law  100-235,  (Computer 
Security  Act  of  1987) 

Directed  NIST  to  recognize  NSA-certified  rating  of  evaluated  trusted 
systems  under  TCSEC  without  requiring  further  evaluation. 

08/90 

U.S.  DoD 

NCSC-TG-01 1,  vl.0,  Trusted  Network  Interpretation  Environments  Guide¬ 
line  - 

•  Guidance  for  Applying  the  TNI,  National  Computer  Security 

Center 

TNI,  Part  of  Rain¬ 
bow  Series 

Red  Book 

1990 

ISO/IEC 

JTC1  SC27  WG3  formed  (Joint  Technical  Committee,  Information  Tech¬ 
nology,  Subcommittee-27  “Security  Evaluation  Criteria.”) 

SC27 

05/90 

European  Com¬ 
munities 

Information  Technology  Security  Evaluation  Criteria  (ITSEC),  vl.0 

•  Commission  of  the  European  Communities  (to  include  Germany, 

France,  The  Netherlands,  United  Kingdom) 

ITSEC 

06/90 

U.S.  DoD 

NCSC-TG-002,  Trusted  Product  Evaluations  -  A  Guide  for  Vendors,  Na¬ 
tional  Computer  Security  Center 

TPEP,  Part  of 
Rainbow  Series 

Bright  Blue  Book 

07/90 

U.S.  National 

Security  Council 

NSD-42,  National  Policy  for  the  Security  of  National  Security  Telecommu¬ 
nications  and  Information  Systems, 

•  Outlines  that  U.S.  Government  capabilities  for  securing  national 

security  systems  against  technical  exploitation  shall  be  main¬ 
tained  or  improved  if  inadequate...  and  that  a  technical  base 
within  the  government  exist  to  achieve  this  security  and  that  ini¬ 
tiatives  with  the  private  sector  to  maintain,  complement  or  en¬ 
hance  that  base  and  to  ensure  information  security  systems  secu¬ 
rity  products  are  available  to  secure  national  security  systems... 

1991 

NIST[CORRECT] 

Originated  concept  of  Trust  Technology  Assessment  Program  (TTAP) 

•  Alternative  approach  to  emerging  Federal  Criteria  for  performing 

evaluations 

TTAP 

03/91 

U.K.  CESG 

UKSP01,  U.K.  IT  Security  Evaluation  Scheme:  Description  of  Scheme, 
Communications  -Electronics  Security  Group 

04/91 

U.S.  DoD 

NCSC-TG-021,  vl.0,  Trusted  DBMS  Interpretation  of  the  TCSEC,  National 
Computer  Security  Center 

Part  of  Rainbow 

Series 

Purple  Book 
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06/91 

European  Com¬ 
munities 

Information  Technology  Security  Evaluation  Criteria — Provisional  Harmo¬ 
nized  Criteria  (ITSEC),  vl.2,  Office  for  Official  Publications  of  the  Euro¬ 
pean  Communities,  Commission  of  the  European  Community 
(DG/XIII/C.4) 

•  Commission  of  the  European  Communities 

•  Originated  from  national  security  certification  criteria  from 
Germany’s  BSI,  France’s  SCSSI  and  the  Netherlands  NLNCSA 

•  Evaluations  paid  for  by  Vendor 

ITSEC 

11/92 

OECD 

Guidelines  for  the  Security  of  Information  Systems,  1992,  Organization  for 
Economic  Cooperation  and  Development 

12/92 

U.S.  NIST  and 

NS  A 

Federal  Criteria  for  Information  Technology  Security,  vl.0,  Vols.  I  and  II 
•  Intended  as  replacement  to  Orange  Book 

Federal  Criteria 

01/93 

Canadian  CSE 

The  Canadian  Trusted  Computer  Product  Evaluation  Criteria  (CTCPEC), 
Canadian  System  Security  Centre,  Communications  Security  Establishment, 

v3.oe 

•  Canadian  Communication  Security  Establishment 

•  Considered  a  combination  of  ITSEC  and  TCSEC  approaches 

CTCPEC 

06/93 

CC  Sponsoring 
Organizations 

CC  Editing  Board  established 

CCEB 

09/93 

European  Com¬ 
munities 

Information  Technology  Security  Evaluation  Manual  -  Provisional  Harmo¬ 
nized  Methodology,  (ITSEM),  Commission  of  the  European  Community 

ITSEM 

12/93 

ECMA 

Secure  Information  Processing  Versus  the  Concept  of  Product  Evaluation, 
Technical  Report  ECMA  TR/64,  European  Computer  Manufacturers’  Asso¬ 
ciation 

ECMA 

TR/64 

04/94 

U.S.  CNSS 

NSTISSP  No.  6,  National  Policy  on  Certification  and  Accreditation  of  Na¬ 
tional  Security  Telecommunications  and  Information  Systems 

Required  that  all  federal  government  departments  and  agencies  estab¬ 
lish  and  implement  programs  that  mandate  certification  and  accredita¬ 
tion  of  national  security  systems 

1996 

ISO 

ISO/IEC  Guide  65,  General  Requirements  for  Bodies  Operating  Product 
Certification  Systems 

1996 

U.S.  Congress 

Information  Technology  Reform  Act  of  1996  (Part  of  the  National  Defense 
Authorization  Act  for  Fiscal  Year  1996) 

•  Requires  that  OMB  oversee  the  development  and  implementa¬ 

tion  of  standards  and  guidelines  pertaining  to  Federal  computer 
systems  by  the  Department  of  Commerce  through  NIST. 

01/96 

CCEB 

Committee  draft  of  Common  Criteria,  1 .0  released 

•  CPCTEC,  ITSEC  and  TCSEC  combined  to  form  version  1 .0  of 

Common  Criteria 

CC 

01/96- 

10/97 

- 

Public  review,  trial  evaluations  of  Common  Criteria 

CC 
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1997 

U.S.  NSA  and 

NIST 

Trust  Technology  Assessment  Program  (TTAP)  established,  along  with 

TTAP  Oversight  Board  and  TTAF  Evaluation  Facilities  (TEF) 

•  Transitioned  commercial  evaluation  program  into  private  sector 

•  Commercial  organizations  allowed  to  conduct  security  evalua¬ 
tions  of  COTS  computer  security  products  using  TCSEC 

•  TTAP  scheme  used  during  the  transition  to  the  NIAP/Common 
Criteria  Evaluation  Scheme  (NIAP/CCEVS) 

TTAP 

10/97 

CCIMB 

Committee  draft  of  CC,  2.0  beta  released  (Interim  Arrangement — Canada, 
United  Kingdom,  United  States) 

CC 

11/97 

CEMEB 

CEM-97/017,  Common  Methodology  for  Information  Technology  Security 
Evaluation,  Part  1 :  Introduction  and  General  Model,  v0.6 

•  Outlined  universal  principles  of  evaluation 

CEM  Part  1 

10/97- 

12/99 

CCIMB  with 

ISO/IEC  JTC 1 

SC27  WG3 

Formal  comment  resolution  and  balloting  conducted  (Full  Arrangement — 
Canada,  France,  Germany,  United  Kingdom,  United  States,  Australia,  New 
Zealand). 

CC 

12/97 

U.S.  DoD 

DoD  Instruction  5200.40,  DoD  Information  Technology  Security  Certifica¬ 
tion  and  Accreditation  Process  (DITSCAP), 

•  Outlines  IT  security  certification  and  accreditation  process 

•  Addresses  integrity  analysis  of  integrated  products  (COTS, 

GOTS  or  Non-Developmental  Item  (NDI)) 

•  Looks  at  system  security  test  and  evaluation 

DITSCAP 

05/98 

U.S.  Office  of  the 

President 

Presidential  Decision  Directive/NSC-63,  Critical  Infrastructure  Protection 

•  Required  that  each  department  and  agency  of  the  Federal  Gov¬ 

ernment  be  responsible  for  its  own  critical  infrastructure,  espe¬ 
cially  its  cyber-based  systems 

1999 

CCIMB 

Committee  draft  of  CC,  2.0  revised  released 

Version  became  known  as  ISO  15408 

CC 

03/99 

U.S.  CNSS 

NSTISSAM  COMPUSEC/1-99,  Advisory  on  the  Transition  From  the 

Trusted  Computer  System  Evaluation  Criteria  to  the  International  Common 
Criteria  for  Information  Technology  Security  Evaluation, 

05/99 

NIST/NSA 

NIAP  Common  Criteria  Evaluation  and  Validation  Scheme  for  Information 

Technology  Security,  (Organization,  Management  and  Concept  of  Opera¬ 
tions)  Scheme  Publication  #1,  v2.0 

•  National  Information  Assurance  Partnership  established  (NIAP) 

•  the  NIAP  is  the  collaborative  effort  of  NIST  and  NSA 

•  Serves  as  interface  to  Common  Criteria 

•  the  NIAP  Validation  Body  provides  technical  guidance  to  testing 
laboratories  and  validates  IT  security  evaluations  for  confor¬ 
mance  to  the  Common  Criteria 

NIAP 

08/99 

CEMEB 

CEM-99/045,  Common  Methodology  for  Information  Technology  Security 
Evaluation,  Part  2:  Evaluation  Methodology,  vl.O 

CEM  Part  2 

12/99 

ISO/IEC 

ISO/IEC  15408,  Information  Technology  -  Security  Techniques  -  Evalua¬ 
tion  Criteria  for  IT  Security,  Parts  1-3  released  (ISO/IEC  SC  27) 

•  First  international  information  technology  security  evaluation 

criteria  standard. 

CC  Parts  1-3 

12/99 

(Ongoing) 

CCIMB 

(Ongoing)  Respond  to  “Requests  for  Interpretations,”  issue  “Final  Interpre¬ 
tations,”  incorporate  Final  Interpretations 

CC 
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01/00 

U.S.  CNSS 

NSTISSP  No.  11,  National  Policy  Governing  the  Acquisition  of  Information 
Assurance  (IA)  and  IA-Enabled  Information  Technology  (IT)  Products, 

•  Required  that  acquisition  of  all  COTS  LA  and  IA-enabled  IT 

products  be  limited  to  those  evaluated  in  accordance  with  the 
Common  Criteria  or  the  NIAP  Evaluation  and  Validation  Pro¬ 
gram  or  the  NIST  FIPS  validation  program 

02/00 

U.S.  CNSS 

NSTISSAM  INFOSEC/2-OO,  Advisory  Memorandum  For  the  Strategy  For 
Using  the  National  Information  Assurance  Partnership  (NIAP)  For  the 
Evaluation  of  Commercial  Off-The-Shelf  (COTS)  Security  Enabled  Infor¬ 
mation  Technology  Products. 

•  Addressed  evaluation  procedures  and  processes 

•  Outlined  that  the  NIAP  would  review  laboratory  report  to  deter¬ 
mine  if  analysis  was  consistent  with  Common  Criteria  require¬ 
ments 

•  Advised  that  government  customers  would  look  to  the  NIAP 
program  for  security  and  security-enabled  COTS  IT  product 
evaluation  requirements 

04/00 

U.S.  CNSS 

NSTISSI  No.  1000,  National  Information  Assurance  Certification  and  Ac¬ 
creditation  Process  (NIACAP), 

•  Provided  guidance  on  how  to  implement  NSTISSP  No.  6 

NIACAP 

05/00 

Multiple 

1 .  Common  Criteria,  Arrangement  on  the  Recognition  of  Common  Criteria 
Certificates  in  the  Field  of  Information  Technology;  (Common  Criteria 
Recognition  Agreement  signed  (Harmonized  Arrangement —  Australia, 
Canada,  Finland,  France,  Germany,  Greece,  Italy,  The  Netherlands,  New 
Zealand,  Norway,  Spain,  United  Kingdom,  United  States,  later  Israel  and 
Sweden) 

•  Sought  to  ensure  evaluations  are  performed  at  consistent  stan¬ 
dards 

•  Improve  availability  of  evaluated  products 

•  Eliminate  burden  of  multiple  evaluations 

•  Continuously  improve  evaluation  process 

•  Merged  members  of  prior  arrangements,  members  of  the  full  CC 
arrangement  with  members  of  the  ITSEC  arrangement 

•  Major  goal  of  CC:  To  work  with  ISO  Joint  Technical  Commit¬ 
tee,  Subcommittee  27  (JTC1  SC27  WG3)  to  make  CC  an  inter¬ 
national  ISO  Standard 

2.  Common  Criteria,  Arrangement  on  the  Recognition  of  Common  Criteria 
Certificates  in  the  Field  of  Information  Technology,  Annex  C 

•  Outlined  requirements  for  certification/validation  body 

CCRA 

05/00 

NIST/NSA 

NIAP  Common  Criteria  Evaluation  and  Validation  Scheme  for  Information 
Technology  Security,  (Validation  Body  Standard  Operating  Procedures), 
Scheme  Publication  #2,  vl.5 

NIAP 

08/00 

NIST/NSA 

NIAP,  NIAP  Common  Criteria  Evaluation  and  Validation  Scheme  for  In¬ 
formation  Technology  Security,  (Guidance  to  Sponsors  of  IT  Security 
Evaluations)  Scheme  Publication  #5,  vl.O 

NIAP 

08/00 

NIST 

NIST,  SP  800-23,  Guidelines  to  Federal  Organizations  on  Security  Assur¬ 
ance  and  Acquisition/Use  of  Tested/Evaluated  Products,  Recommendations 
of  the  National  Institute  of  Standards  and  Technology, 

•  Instructs  federal  agencies  to  give  substantial  consideration  in  IT 

procurement  and  deployment  to  products  that  have  been  evalu¬ 
ated  and  tested  against  security  specifications  and  requirements, 
such  as  NIST  protection  profiles  based  on  the  Common  Criteria 
and  ISO/IEC  15408 

03/01 

NIST/NSA 

NIAP  Common  Criteria  Evaluation  and  Validation  Scheme  for  Information 
Technology  Security,  (Guidance  to  CCEVS  Approved  Common  Criteria 
Testing  Laboratories)  Scheme  Publication  #4,  vl.O 

NIAP 

08/01 

CEMEB 

CEM-2001/0015,  Common  Methodology  for  Information  Technology  Secu¬ 
rity  Evaluation,  Part  2:  Evaluation  Methodology,  Supplement:  ALC  FLR  - 
Flaw  Remediation,  vl.O 

CEM  Part  2 

Supplement 
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2002 

U.S.  Congress 

The  Federal  Information  Security  Management  Act  of  2002,  H.R.  2458 

•  Requires  that  each  agency  conduct  periodic  testing  and  evalua¬ 

tion  of  the  effectiveness  of  information  security  policies,  proce¬ 
dures,  and  practices 

FISMA 

02/02 

NIST/NSA 

NIAP,  NIAP  Common  Criteria  Evaluation  and  Validation  Scheme  for  In¬ 
formation  Technology  Security,  (Guidance  to  Validators  of  IT  Security 
Evaluations)  Scheme  Publication  #3,  vl.O 

NIAP 

10/02 

U.S.  DoD 

DoD  Directive  8500.1,  Information  Assurance, 

•  Required  that  all  IA  or  IA-enabled  hardware,  firmware  or  soft¬ 

ware  products  comply  with  NSTISSP  No.  1 1 

10/02 

NIST 

NIST  SP  800-37,  Guidelines  for  the  Security  Certification  and  Accreditation 
of  Federal  Information  Technology  Systems 

•  Discusses  use  of  standards  such  as  the  CC,  ISO/IEC  1 5408  for 

component  product-level  evaluations,  but  further  suggests  an 
evaluation  of  the  integrated  system  to  ensure  greater  security 

10/02 

U.S.  CNSS 

Annex  A  to  NSTISSP  No.  11,  Deferred  Compliance  Authorizations  (DCAs) 

12/02 

U.S.  Congress 

National  Defense  Authorization  Act  for  Fiscal  Year  2003,  Subtitle  F  - 
Information  Technology,  Section  352,  P.L.  107-314,  (H.R.  4546) 

•  Required  that  acquisition  of  DoD  IA  and  IA-enabled  products  be 

limited  to  evaluated  products 

05/02 

U.S.  DoD/USAF 

AF-CIO  Policy  Memorandum  02-14;  Acquisition  of  Information  Assurance 
(LA)  and  IA-Enabled  Information  Technology  (IT)  Products 

•  Required  that  all  government  acquired  COTS/GOTS  IA  and  IA- 

enabled  products  be  evaluated  against  the  Common  Criteria  or 
the  NIAP  Evaluation  and  Validation  Program  or  the  NIST  FIPS 
validation  program 

02/03 

U.S. DoD 

DoD  Instruction  8500.2,  Information  Assurance  (IA)  Implementation 

•  Required  that  protection  profiles  be  developed  in  accordance 
with  the  Common  Criteria  within  the  NIAP  framework 

•  NS  A  will  generate  protection  profiles 

•  Acquisition  of  GOTS  IT  products  is  limited  to  products  evalu¬ 
ated  by  NS  A,  COTS  IT  products  are  limited  to  those  evaluated 
through  the  Common  Criteria,  the  NIAP  Evaluation  and  Valida¬ 
tion  Program  or  FIPS 

06/03 

U.S.  CNSS 

NSTISSP  No.  11,  National  Policy  Governing  the  Acquisition  of  Information 
Assurance  (IA)  and  IA-Enabled  Information  Technology  (IT)  Products, 
Revised 

•  Set  07/01/02  deadline  for  implementation  of  the  requirement  that 

acquisition  of  all  COTS  IA  and  IA-enabled  IT  products  be  lim¬ 
ited  to  those  evaluated  in  accordance  with  the  Common  Criteria 
or  the  NIAP  Evaluation  and  Validation  Program  or  the  NIST 

FIPS  validation  program 

10/03 

NIST 

NIST  SP  800-36,  Guide  for  Selecting  Information  Technology  Security 
Products,  Recommendations  of  the  National  Institute  of  Standards  and 
Technology 

•  Indicates  that  organizations  should  consider  acquisition  of  IT  se¬ 

curity  products  that  have  been  evaluated  against  specifications 
and  requirements  such  as  protection  profiles  based  on  ISO/IEC 
15408  and  the  Common  Criteria 
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Annex  F  Software  Tools  for  Security  Analysis  and  Pro¬ 
active  Defense 


This  annex  discusses  the  potential  application  of  tools  to  support  security  evaluations.  It 
first  notes  that  most  vulnerabilities  are  caused  by  a  few  well-known  implementation  er¬ 
rors,  and  discusses  other  reasons  why  tools  are  being  increasingly  applied  to  the  problem 
of  developing  secure  software,  for  security  analysis  and  proactive  defense.  This  is  fol¬ 
lowed  by  a  brief  overview  about  tools,  discussion  of  two  particular  types  of  tools  (static 
vulnerability  identification  tools  and  fuzz  testing  tools),  and  a  brief  discussion  of  some  of 
the  many  other  kinds  of  tools  available.  The  next  sections  demonstrate  that  the  CC  does 
not  currently  require  the  tool  use,  and  then  explains  why  tools  (properly  used)  should  re¬ 
duce  vulnerabilities  in  software.  This  annex  concludes  with  a  discussion  about  the  impli¬ 
cations  of  increasing  the  use  of  tools  to  support  evaluations. 


Most  are  vulnerabilities  caused  by  common  well-known  errors 

Most  security  vulnerabilities  are  caused  by  a  relatively  small  set  of  well-known  software 
implementation  errors.  In  the  CERT’s  vulnerability  reports,  9  of  13  advisories  in  1998 
and  at  least  half  of  the  1999  advisories  involved  buffer  overflows.  An  informal  1999  sur¬ 
vey  on  the  Bugtraq  security  mailing  list  found  that  2/3  of  the  respondents  believed  buffer 
overflows  were  the  leading  cause  of  system  security  vulnerabilities  (the  remaining  re¬ 
spondents  identified  “mis-configuration'”  as  the  leading  cause)  [Cowan  1999].  Buffer 
overflows  are  an  old,  well-known  problem,  yet  buffer  overflows  continue  to  resurface 
[McGraw  2000].  [More  CURRENT  REFERENCES  INSERT] 

The  Open  Web  Application  Security  Project  (OWASP)  has  developed  “The  OWASP  Top 
Ten,”  which  represents  a  broad  consensus  of  the  most  critical  web  application  security 
flaws  are.  Their  list  is: 


1 .  invalidated  input, 

2.  broken  access  control, 

3.  broken  authentication  and  session  management, 

4.  cross  site  scripting  (xss)  flaws, 

5.  buffer  overflows,  injection  flaws, 

6.  improper  error  handling, 

7.  insecure  storage,  denial  of  service, 

8.  and  insecure  configuration  management.  [OWASP  2004] 

Gary  McGraw  has  stated  that  the  most  common  reasons  for  vulnerabilities  are: 
1 .  buffer  overflows, 
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2.  race  conditions, 

3.  errors  in  random  number  generation, 

4.  misuse  of  cryptography, 

5.  and  trust  problems  (failing  to  validate  input,  trusting  input  too  much,  and  au¬ 
thentication.  [FESTA200n 

Books  on  developing  secure  software  typically  identify  a  set  of  common  errors  that  pro¬ 
duce  security  vulnerabilities  and  how  to  prevent  them.  [HOWARD2002a]  [VIEGA2002] 
[WHEELER2003] 

The  Common  Criteria’s  own  Common  Evaluation  Methodology  (CEM)  implies  that  cer¬ 
tain  situations  are  leading  causes  of  vulnerabilities.  The  CEM  guidance  for  implementing 
vulnerability  analysis  (AVA  VLA)  has  a  long  list  of  common  vulnerabilities  and  mistakes 
that  “should  be  considered’’  [CC1999] 

Since  a  relatively  small  set  of  vulnerabilities  appear  to  cause  most  of  the  security  vulner¬ 
abilities,  it  is  reasonable  to  ask  if  tools  could  help  find  some  of  them  to  make  the  process 
more  efficient.  There  are  other  reasons  to  consider  tools  as  well. 

Why  Tools? 

Several  other  factors  drive  the  increasing  interest  in  using  tools  to  reduce  vulnerabilities, 
including  the  following: 

1.  Large  and  increasing  code  sizes.  In  1990,  Windows  3.1  was  2.5  million  lines  of 
code;  by  2001  Windows  XP  contained  40  million  lines  of  code.  [FESTA2001] 
Red  Hat  Linux  7.1  included  over  30  million  physical  source  lines  of  code  (SLOC) 
by  2001,  compared  to  well  over  17  million  SLOC  in  version  6.2  of  just  one  year 
earlier.  [WHEELER2002]  These  massive  and  ever-increasing  sizes  make  manual 
review  more  difficult,  and  more  likely  to  miss  problems,  driving  developers  in¬ 
creasingly  towards  using  tools.  Even  in  popular  open  source  software  projects 
where  “all  submitted  code  is  inspected  by  other  members  of  the  group,” 
[JACKSON2004]  . 

2.  Time-to-market  pressures.  In  theory,  given  enough  time,  manual  reviews  could 
thoroughly  examine  any  product.  But  time-to-market  pressures  make  time  a  lux¬ 
ury,  driving  software  development  industry  to  use  tools  where  practical  if  they 
wish  to  reduce  vulnerabilities. 

Tool  Basics 

Many  tools  are  available  that  attempt  to  identify  and/or  counter  previously  unknown  vul¬ 
nerabilities  in  a  program.  Generally  these  tools  can  be  divided  into  static  tools  and  dy¬ 
namic  tools: 

a.  Static  tools  do  not  execute  the  program  being  evaluated.  These  tools  examine  the 
source  code  or  binary  code  of  the  program  to  counter  vulnerabilities.  One  particu¬ 
lar  type  of  static  tool  is  a  “vulnerability  identification  tool,”  which  searches  for 
patterns  that  suggest  vulnerabilities.  Other  tools  can  exploit  annotations  to  prove 
(or  fail  to  prove)  some  property  of  the  program.  Compilers  have  many  built-in 
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checks,  and  often  provide  options  (or  can  have  them  added)  to  impose  stricter  re¬ 
quirements  on  their  inputs;  these  can  be  used  to  require  the  software  to  have  cer¬ 
tain  quality  properties  that  may  reduce  the  likelihood  of  security-related  flaws. 
Techniques  that  prove  that  a  program  has  or  does  not  have  certain  properties,  us¬ 
ing  mathematics  and  certain  assumptions,  also  fit  into  this  category.  A  vast  num¬ 
ber  of  such  tools  require  the  program’s  source  code,  and  cannot  be  applied  effec¬ 
tively  without  it. 

Dynamic  tools  do  execute  the  program  being  evaluated.  These  tools  may  inject 
malicious  input  to  see  what  the  program  does  with  it,  or  manipulate  the  environ¬ 
ment  to  see  if  the  program  can  handle  changes  in  its  environment  well,  or  detect 
situations  strongly  suggesting  a  vulnerability  (and  then  countering  it).  Many  of 
these  tools  can  be  used  without  source  code,  but  a  number  essentially  require 
source  code  because  they  require  the  insertion  of  instrumentation  into  the  source 
code. 

The  literature  generally  suggests  that  the  best  approach  is  to  use  several  different  types  of 
tools  together.  Any  specific  tool  has  many  limitations,  for  example,  a  tool  may  only  find  a 
certain  limited  class  of  vulnerabilities,  it  may  work  only  on  a  limited  set  of  languages,  or 
it  may  only  work  when  examining  software  developed  for  specific  platforms  or  circum¬ 
stances.  Many  tools  have  significant  false  positive  or  false  negative  rates.  As  a  result,  it 
is  often  more  effective  to  use  a  set  of  different  kinds  of  tools,  so  that  each  tool  can  ad¬ 
dress  other  tools’  weak  points. 

Tools  are  not  a  panacea.  Tools  are  best  considered  an  adjunct  to  human  review  to  help 
make  evaluations  more  cost-effective,  instead  of  considering  tools  a  complete  replace¬ 
ment  for  humans.  Other  measures,  such  as  perfonning  human  review  of  the  software  (re¬ 
quirements,  design,  code,  tests)  and  ensuring  that  developers  understand  how  to  develop 
secure  software,  can  help  counter  or  uncover  problems  that  the  tools  cannot.  Neverthe¬ 
less,  tools  can  be  a  very  useful  adjunct  to  human  evaluation.  For  example,  Microsoft’s 
“Trustworthy  Computing  Security  Development  Lifecycle”  includes  both  static  checking 
tools  and  fuzz  testing,  as  well  as  human  review  [LIPNER2005]. 

The  following  sections  describe  in  more  detail  two  specific  kinds  of  tools:  static  vulner¬ 
ability  identification  tools  and  fuzz  testing  tools.  This  is  followed  by  a  brief  discussion  of 
some  of  the  many  other  tools  available.  Specific  tools  are  mentioned  as  examples,  and  are 
not  endorsements  of  any  particular  tool. 

Static  Vulnerability  Identification  Tools 

One  type  of  tool  especially  relevant  to  security  evaluations  are  what  this  report  will  term 
“static  vulnerability  identification  tools.”  These  tools  are  designed  to  examine  source 
code  (or  in  rare  cases,  object  code)  and  identify  patterns  that  suggest  the  presence  of  a 
vulnerability.  This  is  in  contrast  to  approaches  such  as  formal  methods  approaches, 
which  formally  prove  a  particular  property  based  on  static  analysis  (but  require  careful 
statement  of  the  property  to  be  proved  and  assumptions  that  can  be  made). 

A  few  papers  that  examine  static  vulnerability  identification  tools,  some  of  which  also 
examine  other  tools,  include: 
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•  [COWAN2003]  reviews  various  vulnerability  identification  tools  released  under 
an  open  source  software  license,  calling  such  tools  “software  auditing”  tools. 

•  [BROADWELL2002]  used  static  source  code  analysis  and  software  fault  injec¬ 
tion  against  some  commonly-used  applications  and  concluded  that  although  static 
tools  found  many  false  positives,  “when  the  tool  did  find  an  error  [it  was]  ex¬ 
tremely  useful.”  They  also  found  that,  when  comparing  static  and  dynamic  ap¬ 
proaches,  “the  strengths  of  the  two  types  of  tools  can  be  combined  in  mutually 
advantageous  ways.” 

•  [WILANDER2002a]  reviews  some  publicly-available  static  tools  and  argues  that 
tools  to  help  clean  up  vulnerable  code  are  necessary,  but  that  these  tools  should  be 
“a  support  during  development  and  code  auditing,  not  [a]  substitute  for  manual 
debugging  and  testing.”  This  is  because  static  tools  using  lexical  analysis  produce 
too  many  false  positives,  while  other  static  tools  produce  too  many  false  nega¬ 
tives. 

•  [NAZARI02002]  reviews  several  static  analysis  tools,  and  reports  that  they  will 
“never  replace  a  good  manual  audit  of  the  code,”  but  that  such  tools  can  “help  im¬ 
prove  the  state  of  your  code  in  development  or  afterwards...  the  use  of  two  [or 
more]  tools  is  recommended...  [these  tools]  help  assist  you  in  the  auditing  proc¬ 
ess,  not  automate  it.”  [WILANDER2002b]  includes  more  detail. 

•  [TEVIS2004]  notes  that  static  tools  (tenned  code  security  checkers)  provide  an 
excellent  service,  though  they  still  need  to  improve.  Tevis  reported  that  most  of 
the  current  tools  are  limited  to  Unix  (not  Windows  or  Macintosh),  require  a  sig¬ 
nificant  amount  of  expert  knowledge  for  use,  and  that  analysis  is  time-consuming. 
Tevis  claims  that  such  tools  cut  down  only  about  %  to  1/3  of  the  analysis  that 
needs  to  be  performed,  but  does  not  provide  justification  for  these  figures.  Tevis 
argues  developers  should  “move  into  a  functional  [programming]  paradigm”  to 
improve  security,  however,  there  is  little  evidence  that  developers  are  willing  to 
radically  switch  to  such  an  approach,  and  there  is  also  little  evidence  that  this 
would  really  solve  the  problem. 

These  tools  have  both  false  negative  &  false  positive  rates.  They  have  false  negatives 
(there  are  problems  they  cannot  find);  they  will  only  find  those  problems  that  match  pat¬ 
terns  in  their  pattern  database.  To  be  fair,  humans  can’t  guarantee  that  they  will  find  all 
vulnerabilities  either.  Many  of  these  tools  also  have  a  large  false  positive  rate  (they  will 
report  code  instances  as  suspicious  that  are  not  actually  security  vulnerabilities),  though 
there  is  reason  to  believe  these  false  positives  can  be  reduced  as  these  tools  improve  their 
analysis  techniques.  Often  this  is  a  trade-off;  tools  with  fewer  false  negatives  tend  to 
have  more  false  positives,  and  vice  versa.  Thus,  many  of  the  papers  describe  these  tools 
as  aids  to  help  speed  human  evaluation  (by  helping  people  find  the  riskiest  areas  of  the 
software),  instead  of  being  replacements  for  human  evaluation. 

Fuzz  Testing  Tools 

One  approach  for  dynamically  detecting  security  vulnerabilities  is  called  “Fuzzing,”  that 
is,  generating  a  large  number  of  random  test  cases  and  seeing  if  the  program  does  not 
crash  or  hang.  The  original  fuzzing  approach  had  the  following  characteristics  (as  defined 
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at  the  “Fuzz  Testing  of  Application  Reliability”  website  at 
http://www.cs.wisc.edu/~bart/fuzz/fuzz.html): 

b.  “The  input  is  completely  random.  We  do  not  use  any  model  of  program  behavior,  ap¬ 
plication  type,  or  system  description.  This  is  sometimes  called  black  box  testing.  In 
the  original  UNIX  studies  (1990  and  1995),  the  random  input  was  simply  random 
ASCII  character  streams.  For  our  X- Window  study  (in  the  1995  study)  and  our  Win¬ 
dows  NT  study  (2000),  the  random  input  included  cases  that  had  only  valid  keyboard 
and  mouse  events. 

Our  reliability  criterion  is  simple:  if  the  application  crashes  or  hangs,  it  is  considered  to 
fail  the  test,  otherwise  it  passes.  Note  that  the  application  does  not  have  to  respond  in  a 
sensible  manner  to  the  input,  and  it  can  even  quietly  exit. 

As  a  result  of  the  first  two  characteristics,  fuzz  testing  can  be  automated  to  a  high  degree 
and  results  can  be  compared  across  applications,  operating  systems,  and  vendors.” 

This  is  an  extremely  trivial  test  criteria.  Yet  several  papers  demonstrated  that  many  pro¬ 
grams  could  not  even  pass  these  trivial  criteria.  [MILLER1990]  [MILLER1995] 
[FORRESTER2000] 

Although  it  was  originally  conceived  as  a  trivial  measure  of  reliability,  many  observers 
noticed  that  fuzz  testing  tended  to  identify  problems  that  were  also  security  flaws,  such  as 
input  validation  errors  and  buffer  overflows.  Thus,  people  began  to  use  fuzz  testing  spe¬ 
cifically  as  a  security  test.  In  many  cases,  when  used  as  a  security  test,  truly  random  data 
is  not  created  or  is  not  the  only  possibility;  often  fragments  or  random  alternatives  are 
used.  Also,  when  used  for  security,  some  may  not  only  detect  crashes  or  excessive  com¬ 
putation  times;  they  may  also  instrument  try  to  detect  certain  common  indicators  of  vul¬ 
nerabilities  (such  as  unsafe  openings  of  temporary  files  in  a  shared  directory),  or  the  code 
may  be  instrumented  to  detect  some  “should  not  happen”  situations  (and  intentionally 
crash  the  application  if  they  occur).  However,  in  all  cases  fuzz  testing  does  not  attempt  to 
determine  if  a  program  produced  a  “correct”  answer,  merely  that  the  program  did  not 
have  an  obvious  failure. 

For  example,  in  2004,  Michal  Zalewski  developed  in  his  spare  time  a  simple  tool  called 
“mangleme.”  This  tool  generates  “tiny,  razor-sharp  shards  of  malformed  HTML  [the  data 
format  used  by  web  browsers].”  Yet  this  trivial  tool  managed  to  find  security  problems  in 
every  web  browser  it  examined,  [ZALEWSKI2004a]  [ZALEWSKI2004b]  including  the 
one  that  was  the  basis  of  the  W32.Bofra.E@mm  mass-mailing  worm. 
[SYMANTEC2005]  [USCERT] 

Microsoft  defines  fuzzing  as  “structured  but  invalid  [random]  inputs  to  software  applica¬ 
tion  programming  interfaces  (APIs)  and  network  interfaces  so  as  to  maximize  the  likeli¬ 
hood  of  detecting  errors  that  may  lead  to  software  vulnerabilities.”  In  their  approach, 
small  tools  must  be  developed  specifically  for  each  API  and  file  format,  but  these  tools 
are  small  and  easy  to  write.  Microsoft  recently  added  fuzzing  as  a  required  part  of  their 
“security  development  lifecycle,”  and  reports  extremely  encouraging  results  from  its  use. 
[LIPNER2005] 

Traditional  testing  approaches  often  require  that  a  specific  test  case  be  developed  so  that 
the  “correct”  answer  can  be  determined  before  running  the  test.  As  a  result,  relatively 
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few  tests  are  normally  created  for  software  compared  to  the  set  of  possible  program  in¬ 
puts.  In  contrast,  fuzz  testing  does  not  require  knowing  the  correct  answer  (nor  writing  a 
program  to  check  for  correctness).  Thus,  fuzz  testing  can  check  many  more  possible  in¬ 
puts  than  is  possible  in  traditional  testing  approaches.  And  once  a  failure  occurs,  the  data 
that  caused  the  failure  can  be  used  to  identify  the  root  cause. 

There  are  reasons  to  believe  fuzz  testing  will  become  more  effective  in  the  future  for  ini¬ 
tial  versions  of  software,  unless  developers  change  their  development  approaches.  As 
processor  speeds  increase,  and  the  costs  of  processors  go  down  (enabling  more  parallel¬ 
ism  for  the  same  cost),  the  number  of  possible  tests  fuzz  testing  can  perfonn  goes  up.  In 
addition,  as  the  number  of  paths  in  a  program  goes  up  (due  to  its  increasing  size),  the 
number  of  paths  that  may  have  an  error  that  can  be  detected  by  fuzz  testing  goes  up  as 
well. 

However,  fuzz  testing  does  have  quickly  decreasing  returns  after  it  is  first  used  against  a 
given  program.  Once  initial  problems  are  fixed,  in  most  programs  it  becomes  more  and 
more  difficult  to  find  new  vulnerabilities  with  the  technique.  If  developers  know  that 
fuzz  testing  will  occur,  they  often  devise  stronger  input  validation  routines  to  prevent 
most  invalid  data  from  entering  the  rest  of  the  program.  But  these  are  not  problems  per 
se;  in  particular,  any  process  that  encourages  developers  to  strengthen  their  input  valida¬ 
tion  routines  is  likely  to  be  an  improvement. 

Because  fuzz  testing  has  an  extremely  naive  definition  of  “failure,”  there  are  many  secu¬ 
rity  vulnerabilities  it  cannot  detect.  Nevertheless,  there  is  evidence  that  it  can  be  effective 
as  part  of  a  larger  process  for  detecting  security  vulnerabilities. 

Other  Tools 

There  are  a  vast  number  of  tools  related  to  security,  and  more  are  being  developed  all  the 
time.  Here  are  a  few  examples: 

1.  Compiler  warning  flags  and  style  checkers.  Many  compilers  include  built-in  warning 
flags  to  enforce  certain  style  requirements  not  necessarily  required  by  the  language, 
and  there  are  also  separate  tools  that  can  perform  such  checks.  Typically  these  re¬ 
quirements  are  imposed  as  an  effort  to  detect  common  mistakes,  avoid  constructs  that 
are  often  misused  or  hard  to  maintain,  and  to  improve  understandability/reviewability. 
By  enabling  these  options,  developers  can  avoid  some  common  errors  that  in  some 
cases  lead  to  security  vulnerabilities.  In  some  cases  these  checks  are  added  because 
they  often  indicate  common  errors  that  lead  to  security  vulnerabilities,  so  the  bound¬ 
ary  between  these  tools  and  static  vulnerability  identification  tools  is  blurring. 

2.  Secure  libraries.  Since  easily  made  programming  mistakes  are  the  cause  of  many 
security  vulnerabilities,  one  approach  is  to  modify  existing  programming  libraries  or 
to  create  new  libraries  that  are  easier  to  use  securely.  For  example,  ISO  has  begun 
work  on  a  technical  report  to  define  new  library  functions  for  the  C  programming 
language  to  simplify  development  of  secure  programs. 

3.  Languages  with  improved  security  properties.  The  highly  popular  C  and  C++  pro¬ 
gramming  languages  are  extremely  pennissive,  and  require  programmers  to  perform 
many  low-level  tasks  such  as  tracking  memory.  Mistakes  in  doing  so  cause  many 
problems;  for  example,  C  and  C++  are  the  only  languages  in  widespread  use  where 
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buffer  overflows  can  occur  by  default.  Developers  could  choose  to  use  languages 
where  many  common  mistakes  are  not  possible  or  far  less  likely  can  reduce  the  num¬ 
ber  of  security  vulnerabilities.  However,  no  language  can  prevent  all  possible  secu¬ 
rity  vulnerabilities.  It  is  unlikely  that  a  blanket  requirement  to  avoid  permissive  lan¬ 
guages  would  be  commercially  viable,  especially  if  it  were  applied  at  EAL  5  or  be¬ 
low;  there  is  simply  trillions  of  dollars  invested  in  C/C++  programs,  and  the  expense 
of  rewriting  them  would  be  difficult  to  justify.  Also,  applications  almost  always  run 
on  some  C  and/or  C++  code,  even  if  the  application  itself  is  not  written  in  them,  be¬ 
cause  many  critical  reused  libraries,  nearly  all  language  run-times,  and  nearly  all 
commercial  operating  system  kernels  are  written  in  C.  Sometimes  another  language’s 
run-time  inhibits  some  protective  measures  for  C.  For  example,  the  GNAT  compiler 
(a  popular  Ada  compiler)  uses  “trampolines”  in  its  implementation,  and  must  turn  off 
certain  protections  used  by  some  [Kleen  2004] 

4.  Run-time  environment  countermeasures  for  common  vulnerabilities .  In  some  cases  a 
platform  (such  as  the  operating  system  or  language  run-time)  can  detect  the  symp¬ 
toms  of  a  vulnerability  or  attack,  and  reduce  the  damage  it  can  cause.  Since  buffer 
overflows  are  an  especially  common  security  vulnerability,  many  tools  have  been  de¬ 
veloped  to  detect  and  counter  them  at  run-time  by  halting  the  program  (turning  a  po¬ 
tential  complete  take-over  of  a  machine  into  a  denial-of-service  attack).  Examples  in¬ 
clude  StackGuard  [COWAN1998],  Microsoft’s  /gs  compiler  switch  [BRAY2002], 
IBM’s  Scientific  Subroutine  Package  (SSP)  [WILANDER2003],  and  Red  Hat  Linux’s 
ExecShield  [VANDEVEN2005],  Environmental  countenneasures  have  been  devel¬ 
oped  to  counter  other  vulnerabilities  as  well,  such  as  for  temporary  file  race  condi¬ 
tions  [COWAN2001a],  format  string  errors  [COWAN2001b],  and  double-free  errors. 

5.  Proof-making/checking  tools.  Over  several  decades  there  have  been  many  efforts  to 
develop  tools  to  support  formal  methods.  Proof-checking  tools  can  confirm  the  valid¬ 
ity  of  a  proof,  and  proof-making  tools  can  develop  some  proofs  automatically  (in 
practice,  often  requiring  human  guidance).  In  general,  applying  these  tools  to  source 
code  requires  extreme  rigor,  specialized  language  subsets,  and  a  commitment  to  de¬ 
veloping  source  code  and  the  necessary  annotations  for  proof  simultaneously. 

6.  Standard  test  suites  and  vulnerability  scanners.  Standard  sets  of  tests  can  be  devel¬ 
oped  for  common  product  classes  (such  as  firewalls  or  operating  systems).  Indeed, 
network  security  scanning  tools  (such  as  Nessus)  that  can  actually  send  real  attacks 
(not  just  check  version  numbers)  already  embody  a  large  set  of  specific  security  tests, 
and  can  be  used  to  determine  if  a  product  can  withstand  attacks  that  have  worked 
elsewhere.  Such  test  suites  can  send  attacks  that  have  succeeded  in  the  past,  as  well 
searching  for  general  issues  such  as  unexpected  open  ports  or  cleartext  passwords. 
Such  tools  are  especially  useful  for  regression  testing,  for  example.  They  are  limited, 
obviously,  to  the  specific  items  they  test  for,  so  they  should  be  supplemented  with 
other  tools  designed  to  detect  previously  unknown  kinds  of  vulnerabilities. 

In  some  cases,  it  would  be  possible  to  impose  support  for  certain  tools  in  a  Protection 
Profile.  For  example,  operating  systems  could  be  required  to  include  as  buffer  overflow 
run-time  environment  countermeasure  (which  could  be  checked  using  simple  a  simple 
standard  test  suite),  and  applications  could  be  required  to  enable  certain  compiler  options 
under  certain  conditions. 
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Common  Criteria  does  not  require  tools 

A  careful  analysis  of  the  Common  Criteria  shows  that  while  it  permits  the  use  of  tools,  it 
does  not  require  the  use  of  any  tool.  In  addition,  the  Common  Criteria’s  approach  to 
source  code  inhibits  the  use  of  many  tools;  full  source  code  is  only  required  at  EAL  5, 
and  none  is  required  below  EAL4  in  the  current  version. 

The  Common  Criteria  does  have  some  requirements  for  vulnerability  analysis  in  the  fam¬ 
ily  AVAVLA.  The  lowest  level,  AVAJVLA.l,  is  required  at  EAL2;  the  next-lowest 
level,  AVA_VLA.2,  is  required  at  EAL4.  However,  the  instructions  for  performing  such 
evaluations  are  vague,  and  do  not  specifically  require  any  kind  of  tool  use,  even  when 
such  tools  are  available  and  appropriate.  A  vulnerability  analysis  is  defined  by  the  CEM 
as  a  “systemic  search,”  (section  8.10.3.2).  The  EAL4  CEM  work  unit  4:  AVA_VLA.2-9 
provides  a  list  of  issues  that  an  evaluator  should  consider.  However,  the  entire  CEM  text 
appears  to  presume  a  manual  consideration  of  issues,  and  manual  creation  of  tests  to 
prove  or  disprove  the  existence  of  a  vulnerability.  It  certainly  does  not  require  the  use  of 
tools. 

At  the  higher  levels,  some  of  the  CC  text  could  be  interpreted  as  supporting  the  use  of 
tools,  but  it  does  not  explicitly  require  them.  The  higher-level  AVA_VLA.3  (required  at 
EAL5)  requires  a  “systematic”  search  for  vulnerabilities;  a  process  using  tools  might  be 
tenned  systematic,  but  a  manual  systemic  process  could  also  meet  this  requirement  so 
there  is  no  clear  requirement  for  them.  The  strongest  vulnerability  analysis  requirement, 
AVA_VLA.4,  is  required  for  EAL  6  and  7;  it  requires  a  justification  that  the  analysis 
“completely  addresses  the  TOE  deliverables”  but  again  does  not  require  the  use  of  tools. 

Since  some  types  of  tools  tend  to  report  a  number  of  false  positives,  evaluators  may  have 
a  financial  disincentive  to  use  tools  to  examine  potential  vulnerabilities.  An  evaluator  can 
increase  profits  by  merely  positing  a  small  set  of  vulnerabilities  (as  pennitted  by  the  CC), 
so  that  only  a  limited  subset  of  vulnerabilities  needs  to  be  considered,  instead  of  using 
tools  as  a  supplement  to  their  analysis. 

In  many  evaluations,  the  lack  of  source  code  severely  restricts  tool  use,  since  many  tools 
require  access  to  the  source  code  and/or  the  ability  to  rebuild  the  program.  In  the  Com¬ 
mon  Criteria,  evaluations  at  EAL  3  and  below  do  not  require  source  code,  and  at  EAL4 
only  a  subset  of  source  code  is  available  (as  requirement  AD V  IMP.l).  Access  to  all  of  a 
program’s  source  code  is  not  required  until  EAL  5  (as  requirement  ADV  IMP.2).  Since 
many  evaluations  only  occur  at  EAL4  and  below  (to  meet  mutual  arrangement  require¬ 
ments  as  well  as  to  reduce  costs),  the  current  CC  structure  makes  it  difficult  to  employ 
tools  for  most  evaluations.  Even  if  the  vendor  is  willing  to  release  all  their  source  code  to 
an  evaluator  (a  common  circumstance  as  long  as  protective  measures  are  put  in  place), 
evaluators  cannot  consider  the  whole  set  since  to  do  so  would  create  an  unfair  situation 
between  vendors.  It  is  expected  that  the  next  revision  of  the  Common  Criteria  will  require 
all  source  code  at  EAL4,  but  this  does  not  help  in  lower-level  evaluations.  See  the  discus¬ 
sion  on  source  code  review  for  more  information. 

This  is  more  striking  when  the  actual  Common  Criteria  requirements  are  compared  with 
people’s  expectations.  All  stakeholder  classes  expected  that  the  NIAP  would  provide 
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tools  and  require  source  code  analysis  (see  section  5,  “Testing  of  Products  in  Evalua¬ 
tion”). 


Tools  should  reduce  vulnerabilities  and  effort  to  find  them 

Tools,  when  properly  developed  and  used,  should  detect  and/or  counter  a  significant  pro¬ 
portion  of  the  most  common  vulnerabilities.  As  noted  earlier,  most  vulnerabilities  are 
caused  by  a  small  set  of  common  development  errors;  implementing  tools  specifically  to 
counter  those  errors  should  reduce  vulnerabilities,  particularly  at  the  lower  levels  of  as¬ 
surance.  Vulnerability  identification  tools  are  focused  on  finding  common  causes  of  vul¬ 
nerability,  and  fuzz  testing  tools’  approach  also  tends  to  find  security  vulnerabilities.  In 
short,  employing  tools  to  focus  on  likely  problems  should  significantly  reduce  the  num¬ 
ber  and  severity  of  vulnerabilities. 

There  is  relatively  little  data  on  the  speed  or  manpower  required  for  tool-assisted  evalua¬ 
tions  as  compared  to  manual  evaluations.  It  is  reasonable,  however,  to  presume  that 
evaluations  supported  by  tools  should  require  less  manpower  than  a  purely  manual  ap¬ 
proach.  Secure  Software,  a  company  that  performs  tool-assisted  security  evaluations,  re¬ 
ports  an  estimate  of  3,000  lines  of  code  per  hour  reviewed  and  analyzed  when  they  use 
their  (in-house)  tools,  versus  100  lines  of  code  per  hour  if  done  manually  (for  what  they 
believe  are  similar  levels  of  scrutiny)  [SECURE2004]. 

[VANDEVEN2005]  reports  on  the  experience  of  employing  a  single  run-time  counter¬ 
measure;  they  found  that  in  the  period  from  November  1,  2003  to  August  11,  2004,  there 
were  16  security  issues  with  more  severity  than  a  Denial  of  Service  problem  and  for 
which  an  exploit  was  made  available.  Out  of  these  16  exploits,  12  were  countered  by 
their  countermeasure,  yielding  a  success  rate  of  75%  against  unknown  vulnerabilities  (re¬ 
ducing  their  risk  and  impact). 

Researchers  are  already  working  to  improve  the  results  of  such  tools.  For  example, 
[DACOSTA2003]  found  that  that  most  vulnerabilities  are  clustered  near  inputs,  a  plausi¬ 
ble  hypothesis  implied  (but  not  proven)  by  previous  tool  developers’  work.  Thus,  a  tool 
that  raises  the  risk  level  based  on  nearness  to  inputs  should  correctly  identify  what  is  the 
riskiest. 


Implications 

Tools  are  available,  but  the  Common  Criteria  does  not  currently  require  their  use.  As  a 
result,  tools  are  often  not  used  or  required,  even  when  it  would  be  sensible  to  do  so. 

Tools  could  be  used  during  the  evaluation  process  itself,  e.g.,  to  perform  source  code 
scanning,  fuzz  testing,  and  standard  test  suites.  However,  this  raises  a  fundamental  ques¬ 
tion:  How  will  these  tools  be  developed  and  deployed?  There  are  several  options  for  de¬ 
velopment  in  each  tool  category,  if  they  are  to  be  used: 

1 .  Select  a  specific  commercial  product  for  use.  This  has  the  advantage  of  simplicity 
and  commercial  support.  However,  if  it  is  a  commercial  product,  doing  so  will  put 
competing  products  at  a  significant  disadvantage,  and  any  such  selection  is  likely 


F-9 


to  be  challenged.  In  practice,  the  costs  of  such  products  may  be  quite  large  (espe¬ 
cially  since,  as  a  monopoly  supplier,  a  vendor  may  take  advantage  of  their  status 
where  they  can).  Any  required  tool  essentially  becomes  part  of  a  government 
mandate  &  a  government  regulation  for  production.  Note  also  that  vendors  of 
such  products  tend  to  not  reveal  in  detail  their  measurement  criteria.  As  a  result, 
this  option  would  essentially  cede  the  definition  of  security  to  a  third-party  com¬ 
mercial  firm.  This  would  also  make  it  more  difficult  for  firms  to  perform  such 
testing  ahead-of-time,  since  such  products  may  be  as  expensive  as  a  Common  Cri¬ 
teria  evaluation. 

2.  Select  a  set  of  commercial  products  for  use.  This  option  avoids  putting  competing 
products  at  a  significant  disadvantage.  However,  this  also  means  that  the  meas¬ 
urement  criteria  will  vary,  depending  on  which  product  is  used,  greatly  reducing 
the  uniformity  desired  for  the  Common  Criteria.  And  again,  this  option  essen¬ 
tially  cedes  the  definition  of  security  to  third  parties. 

3.  NIAP-developed  tools.  The  original  plans  for  the  NIAP  included  the  intent  to  de¬ 
velop  tools,  which  would  counter  the  disadvantages  of  the  first  two  approaches. 
The  disadvantage  here  would  be  the  costs  of  tool  development  and  maintenance. 
On  the  positive  side,  the  NIAP  could  freely  release  its  own  tools,  greatly  increas¬ 
ing  the  likelihood  of  widespread  adoption  (eliminating  product  vulnerabilities 
long  before  they  entered  an  evaluation,  if  they  ever  did).  This  would  also  be 
clearly  fair  and  consistent.  There  are  various  cost-sharing  methods  that  could  be 
used  to  reduce  somewhat  the  costs  of  initial  development,  and  it  is  worth  noting 
that  evaluation  costs  are  simply  hidden  in  product  costs  (so  the  government  would 
pay  at  least  some  money  for  the  previous  options  too).  Commercial  vendors 
would  be  free  to  exploit  those  tools  or  their  ideas,  and  could  also  develop  tools 
that  went  beyond  the  NIAP  tools. 

To  be  practical,  widely  using  tools  ay  require  modifying  lower  EAL  levels  to  require 
source  code.  Many  tools  require  source  code,  and  a  significant  number  require  the  ability 
to  rebuild  an  executable  program  from  source  code  (to  perform  instrumentation).  The  cur¬ 
rent  CC  requires  all  source  code  at  EAL  5,  with  only  a  sample  at  EAL4,  and  nothing  be¬ 
low  EAL4.  The  updated  CC  is  expected  to  require  all  source  code  at  EAL4,  but  nothing 
below  that.  An  alternative  would  be  to  require  all  source  code  (with  build  instructions)  at 
EAL  2  or  3.  This  will  require  intellectual  property  protections,  but  labs  already  make 
such  arrangements  for  EAL4  and  above,  and  there  is  little  evidence  that  vendors  have 
trouble  with  this.  Vendors  can  choose  which  lab  they  believe  will  provide  adequate  pro¬ 
tection  for  their  property,  and  avoid  those  labs  whose  procedures  are  inadequate.  In  the 
end,  it  is  the  code  that  is  executed,  not  documentation,  and  many  customers  are  skeptical 
of  evaluation  processes  that  ignore  the  program  actually  being  executed.  Tools  could  en¬ 
able  at  least  partial  evaluation  of  the  actual  code  that  is  being  executed. 

Of  course,  the  mere  existence  of  tools  and  elimination  of  roadblocks  is  not  enough;  tools 
are  not  relevant  unless  they  are  used.  Tools  could  be  implemented  in  several  ways: 

1.  The  Common  Criteria’s  testing  (ATE)  and  vulnerability  assessment  (AVA)  assur¬ 
ance  classes  could  be  modified  (or  interpreted  by  the  NIAP)  to  specifically  require 
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certain  kinds  of  tool  use  in  certain  circumstances  (e.g.,  for  certain  product  types 
and  EALs). 

2.  The  entry  criteria  for  evaluation  could  be  modified  to  require  the  use  of  certain 
kinds  of  tools  that  reduce  the  likelihood  of  vulnerabilities,  especially  at  higher 
levels  of  assurance.  These  include  build  mechanisms  (e.g.,  compiler  options  to 
detect  or  counter  problems),  environmental  requirements  (e.g.,  buffer  overflow 
protections),  the  use  of  certain  kinds  of  secure  libraries,  and  so  on. 

PPs  of  some  platforms  could  be  modified  to  include  mechanisms  that  reduce  vulnerabili¬ 
ties  of  applications  that  run  on  those  platforms.  For  example,  operating  system  PPs  could 
be  modified  to  require  support  for  a  buffer  overflow  protection  mechanism. 
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letter  of  partnership 

NATIONAL  SECURITY  AGENCY 
AND  NATIONAL  INSTITUTE  OF  STANDARDS 
AND  TECHNOLOGY  I 
AUGUST  22, 1997 


Consumers,  in  both  the  private  and  public  sectors,  need  confidence  and 
assurance  in  the  products  they  use  to  secure  valuable  information.  That 
confidence  is  bolstered  when  the  products  have  been  evaluated,  tested  and 
certified  by  independent  organizations.  As  security  products  change  to  stay 
ahead  of  evolving  threats,  so  must  the  tests,  methods  and  metrics  used  to 
evaluate  them.  The  National  Information  Assurance  Partnership  will  employ  the 
latest  techniques  to  develop  specification-based  tests,  methods  and  tools  so  that 
testing  laboratories  and  certificate  issuing  organizations— as  well  as  consumers 
and  producers  of  information  technology  products  -will  have  objective  measures 
for  evaluating  quafity  and  security. 

Security  product  manufacturers  and  service  providers  stand  to  benefit 
from  having  U.S.  products  mutuaHy  recognized  around  the  world.  When  testing 
labs  in  several  nations  use  common,  yet  stringent,  criteria  to  evaluate  products, 
international  acceptance  follows.  This  partnership  w#  work  closely  with  the  NIST- 
administered  National  Voluntary  Laboratory  Accreditation  Program  to  develop 
technical  criteria  that  NVLAP  wiS  use  when  it  accredits  testing  labs. 

The  goal  of  the  National  Information  Assurance  Partnership  is  to  boost 
consumers'  confidence  in  information  security  products  and  enhance  the  United 
States'  ability  to  gain  international  recognition  and  acceptance  for  products  such 
as  firewalls,  access  control  mechanisms  and  other  U.S.  products  that  help; 
ensure  the  security  of  information  technology  systems  and  networks.  More  :• 
specifically,  N1AP  seeks  to:  1)  promote  demand  and  investment  in  secure 
products;  2)  transition  current  evaluation  and  testing  efforts  from  the  government 
to  private  commercial  labs;  ana  3)  foster  research  an|d  development  in  security 
tests,  methods  and  metrics. 
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As  partners  in  this  endeavor,  we  pledge  to  help  enhance  the  quality  of 
U.S.  information  security  products  and  expand  the  market  for  such  products. 


iLjl:  ujJ 

Wakid  V 


Shukri  Wakid 
Director, 

Information  Technology  Laboratory 
National  Institute  of  Standards  and  Technology 


i  Thomas  McDermott 
De^ity  Director. 

Information  Systems  Security 
National  Security  Agency 
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TERMS  OF  REFERENCE 
FOR  THE  NATIONAL  INFORMATION  ASSURANCE  PARTERNERSHJP 
between  the 

National  butitstc  of  Stiwlards  and  To 
a«l  the 

Nitmul  Seen  rity  Agency 
Information  Systems  Scarify  Orpoiaitioos 


L  Purpose 

Tbe  purpose  of  these  Terms  of  Reference  is  to  establish  the  National  IirfbnmtioQ  Assurance  Partnership 
(NIAP)  between  (be  Information  Technology  Laboratory  (TTL)  at  tbe  National  I  naitutr.  of  Standards  and 
Technology  (NIST),  and  the  Information  Systems  Security  OganizaDoa  at  tbe  National  Security  Agency 
(NS A).  These  Terms  of  Reference  will  be  used  as  tbe  basis  for  a  working  relationship  between  dements 
in  bods  organizations  to  facilitate  wort  oq  information  Technology  security  testing,  assaraoce,  and 
evaluation  issues. 


EL  Responsibilities 
A  NIAP 

Tbe  NIAP  is  a  joint  effort  of  NIST  and  NSA  to  provide  technical  leadership  in  tbe  research  ami 
development  of  seenrity-fdated  Information  Technology  (IT)  test  metlwjds  and  assurance  techniques.  Tbe 
NIAP  win  also  facilitate  the  establishment  of  programs  intended  to  assess  conformance  and  assurance  of 
products  to  security  specifications  or  standards.  Test  methods  will  be  developed  through  collaborative 
research  efforts  with  formal  standards  organizations,  industry  groups,  user  groups,  vendor  associations, 
professional  organizations,  and  oser/applicatvon  affinity  groups 

NIST  and  NSA  have  in  marrd  the  NIAP  to  respond  to  common  govern  a  eat  needs  for  TT  products  and 
systems  chat  have  been  tested  for  conformance  and  assurance  of  their  security-related  properties  to 
specifications  or  standards.  The  goal  of  using  commercial-off-the-shelf  [COTS)  products  to  tbe  maximum 
extent  in  Federal  systems  most  be  balanced  against  cost  and  timeliness  of  testing  the  security  features  of 
those  products.  This  concern  is  shared  by  private  industiy  as  well.  The  INIAP  wifi  remedy  the  lack  of 
existing  research  into  security  rests  and  methods  (ocher  chan  tradition. il  government^ -operated  security 
evaluation  programs). 


Tbe  responsibilities  outlined  in  these  Terms  of  Reference  do  not  preclude 
conducting  research,  testing,  or  evaluations  independently  or  from  seeing 
under  national  policies  and  federal  law. 


either  organization  from 
standards  or  meet  obligations 


B.  NIST  and  NSA  Senior  Management  Representatives 

NIST  and  N$A  will  designate  Management  Representatives  (MRs)  to  provide  guidance,  direction,  and 
prioritizing  to  the  NIAP.  As  representatives  of  Senior  Management  for  ioth  organizations  the  MR  roles 
and  responsibilities  include  the  following: 


•  Tbe  MRs  provide  direction  and  priorities  for  NIAP  projects. 

•  The  MRs  must  jointly  agree  on  activities  designated  as  NIAP  projects.  Neither  MR  may  designate  an 
activity  as  a  NIAP  project  without  the  cx>ncurrenoc/coordi nation  of  tbe  other  MR. 
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•  The  MRs  retain  the  reqxxisibility  and  authority  to  commit  their  organizations'  resources  as  needed  on 
a  project  by  project  basis  Overall  staffing  of  NIAP  projects  will  beat  a  MR-agreed  to  level  of 
support. 

C  NIAP  Management 

The  NIST  and  NSA  MRs  will  each  designate  a  Program  Manger  (PM)  to  foster  the  operational 
relationship  between  both  organizations.  Management  responsibilities  will  include  tbc  following: 

•  The  PMs  ueptemem  HEAP  projects  based  on  direetkn  and  prioriiie|  bom  MRs.  The  official  list  of 
WAP  projects  is  maintained  by  tbc  NIST  PM  as  NlAP  Prefect  Management  Plans , 

.  The  PMs  ensure  that  project  directions  are  consrstea  with  each  respective  organization's  strategic 
plans. 

•  The  PMs  will  jointly  prepare  and  agree  to  project  plans  to  ensure  that  proposed  tasks  and  deliverables 
are  dearly  defined  and  responsive  to  each  respective  organization's  requirements. 

OL  Working  Methods 

Each  organization  will  have  an  equal  voice  (one  voter  per  organization)  in  all  aspects  of  NIAP  decision¬ 
making.  inrtnding  the  selection  of  NIAP  projects,  allocation  of  NIAP  resources,  oversight  of  contractor 
support,  and  technical  direction. 

Recognizing  that  NIST  and  NSA  will  not  have  equality  in  the  amount  of  discretionary  resources 
applicable  to  NIAP.  each  organization  will  first  look  internally  for  resour  as  and/or  contractual  vehicles  to 
aoonmpli'ih  a  given  task.  If  necessary,  it  is  agreed  that  one  organization  i  nay  transfer  funds  to  the  other 
organization  for  the  purpose  of  supporting  a  specific  NIAP  tasking  through  svpvmre  implementing 
agreements  trader  the  Economy  Act  (31  U.S^C.  1535).  This  funding  may  be  used  for  in-house  or 
contractor  support  Each  organization  will  have  access  to  eontnctois  working  on  NIAP  tasking, 
regardless  of  whose  contractor  vehicle  was  used,  to  discuss  project  issues  and  technical  solutions,  but 
under  no  drenmaanecs  should  conflicting  or  contradictory  directions  be  issued  to  the  contractor. 

If  at  any  tune  in  an  NIAP  project,  a  MR  determines  that  the  project  no  longer  meets  organizations  needs, 
the  project  will  be  removed  bom  the  NIAP.  Funding ejtchangodhetwernetie  organizations  well  be 
returned  at  the  discretion  of  the  original  funding  organization.  Either  organization  may  continue  to  fund 
the  project  aad  assume  full  responsibility  for  the  project. 

Project  plans  will  be  developed  for  each  NIAP  Project  and  a  Project  Manager  assigned.  The  project 
manager  will  work  with  the  NIAP  Program  Managers  to  ensure  satisfactory  implementation  of  the 
project's  goals. 

Documents,  working  drafts,  and  working  groups  meeting  minutes  will  nn.  finely  be  made  available  ro  the 
public  when  requested  with  the  exception  of  portions  of  the  project  plans  Chat  are  marked  POUO  or 
Proprietary.  For  documents  requiring  public  comment,  the  operating  method  will  be  ro  seek  maximum 
pubbe  input  with  a  90 day  review  and  comment  period.  Public  pronouncement,  briefings,  and  meetings 
shall  be  jointly  coordinated  and  agreed  to  by  the  MRs  or  PMs  and  Public  Affairs  Offices  (when 
appropriate)  before  announcement  are  made. 


IV.  Points  of  Contact 
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The  NIST  point  of  contact  is  Dr.  Smart  Kalzke.  Infonnatioo  Technology  Laboratory.  (JO  I)  97S29J4 
The  NSA  point  of  coooct  is  Mr.  Louis  Giles.  Commercial  Solutions  anil  Enabling  Technologies.  (4101 
859-6281.  .  1 


V. 


Effective  Date  and  Termination 


These  Terms  <rf  Reference  shall  be  effective  on  the  date  of  the  last  sigi  attire  and  will  be  jointly  reviewed 
annually  by  NIST  and  NSA.  It  may  be  modified  or  rescinded  by 


terminated  unilaterally  upon  written  noooe  by  either  patty  to  the  other  via  certified  raaiL 


^L: 


SHUKRI  A.  WAKff) 

Director 

Inlormattou  Technology  Laboratory 
National  Institute  of  Standards  and  Technology 

DATE:_  Chsltf 


a&reemeru  in  writing  or 


MICHASj.  JACOBS 
Deputy  Director 
Idormabofi  Systems  Security 
Habonil  Security  Agency 


DATE:  & 
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Annex  H  Alternate  Forms  of  Assurance 

Several  interviewees  believed  that  alternative  assurance  methods  are  needed,  es¬ 
pecially  to  reduce  costs  (such  as  a  “CC  lite”).  This  is  the  case  for  organizations  or  situa¬ 
tions  that  cannot  afford  to  pay  for  evaluations,  such  as  many  small  web  applications, 
small  businesses,  and  open  source  software  (OSS)  projects.  Support  for  alternative  assur¬ 
ance  levels  was  strongest  for  use  in  lower  assurance  evaluations.  Many  believed  that  the 
NIAP  evaluation  would  be  strengthened  if  the  alternative  assurance  methods  were  used  to 
supplement  the  NIAP  evaluation,  with  SSE,  CMM  ,  and  CMMI®  specifically  mentioned 
as  examples  of  alternative  assurance  methods. 

H.l  Other  Assurance  Methods 

Examining  documentation,  along  with  some  vulnerability  analysis  and  functional  testing, 
is  certainly  not  the  only  way  to  gain  assurance  that  a  product  is  unlikely  to  contain  vul¬ 
nerabilities  of  certain  kinds.  Other  aspects  could  be  examined  instead  or  in  addition: 

H.1.1  Source/Binary  Code  Review. 

These  reviews  could  be  manual  or  with  tools. 

H.1.2  Code  Proofs. 

This  area  needs  research  in  fonnal  methods. 

H.1.3  Peer  Review/Focused  Code  Review. 

This  is  one  aspect  of  an  overall  development  process  evaluation. 

H.1.4  Development  process. 

This  examines  all  aspects  of  the  development  process,  including  quality  assurance,  defect 
tracking,  etc. 

H.1.5  Standard  Security  Test  Suites. 

For  many  common  application  areas,  such  as  firewalls  and  intrusion  detection  systems,  it 
would  be  possible  to  create  a  standard  security  test  suite  for  each  area.  For  example;  for 
firewalls  it  would  be  possible  to  create  a  standard  set  of  tests  that  any  firewall  should 
withstand.  One  challenge  is  that  such  a  test  suite  must  be  under  constant  improvement 
itself,  since  attackers  continuously  create  new  attacks. 

H.l. 6  Field  Use  with  Few  Reported  Vulnerabilities. 

Although  it  is  not  a  perfect  indicator,  few  reported  vulnerabilities  on  a  product  that  has 
significant  field  use  could  provide  some  assurance.  However,  this  presumes  there  are 
people  who  are  searching  for  vulnerabilities,  that  those  vulnerabilities  are  reported,  and 


The  CMMI  and  Capability  Maturity  Model  Integration  are  registered  trademarks  of  Carnegie  Mellon  Uni¬ 
versity. 
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that  those  vulnerabilities  are  eventually  acknowledged  publicly.  None  of  these  assump¬ 
tions  are  always  true. 

H.1.8  Other  Evaluation  Processes. 

Note  there  is  already  an  evaluation  process  for  cryptographic  modules  (FIPS- 140).  Other 
evaluation  processes  include  DCID  6/3  PL5  and/or  DITSCAP  (soon  to  become 
DIACAP). 

H.2  Other  Constraints/Requirements: 

Other  constraints/requirements  may  include  the  following: 

H.2.1  No/little  vendor  money. 

Many  small  businesses  and  open  source  software  projects  cannot  afford  an  independent 
evaluation  as  structured  today.  It  may  be  valuable  to  devise  assurance  or  evaluation 
processes  that  can  be  used  when  there  is  little  or  no  money  available,  though  the  vendor 
does  have  some  time  available  for  some  kind  of  low-assurance  evaluation. 

H.2.2  Potential  for  malicious  developer  or  vendor. 

The  CC  evaluation  process  as  currently  designed  presumes  there  are  no  malicious  devel¬ 
opers.  For  example,  a  vendor  is  allowed  to  detennine  what  evidence  is  given  to  an 
evaluator;  if  an  evaluator  is  given  false  information,  he  may  reach  a  false  conclusion. 

H.3  Combinations: 

These  issues  and  approaches  can  be  combined  in  many  ways. 

Example  1: 

Use  a  specified  set  of  tools  to  search  for  the  most  common  vulnerabilities 
in  the  source  code,  fixing  problems  found 

Use  a  standard  test  suite  (provided  by  the  evaluation) 

Use  a  CM  process  with  a  few  simple  requirements,  (e.g.,  limiting  who  can 
make  changes  to  the  trusted  product  to  authorized  users  and  ensuring  that 
users  know  exactly  what  they  received) 

Use  a  trusted  delivery  process  including  digitally  signed  executables 

Evaluators  could  briefly  check  to  ensure  that  these  were  done,  taking  no 
more  than  a  few  hours. 

Example  2: 

A  different  high-assurance  evaluation  might  include  these  kinds  of  re¬ 
quirements: 

Proof  of  correctness  at  the  source  code  level 

Peer  review  of  all  source  code  (at  the  line-by-line  level) 

8-hour  developer  training  in  developing  secure  software  (not  including 
formal  methods,  which  would  be  handled  separately),  including  require¬ 
ments,  design,  implementation  issues  (including  common  implementation 
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mistakes),  and  testing.  This  training  must  occur  before  the  developer  is  al¬ 
lowed  to  create  artifacts  for  the  project. 

The  Common  Criteria  could  be  modified  to  include  other  assurance  classes,  even  if  they 
are  not  a  part  of  EALs.  This  would  at  least  increase  understanding  and  it  might  also  en¬ 
courage  use  of  other  assurance  approaches  where  they  are  appropriate. 

However,  one  general  complication  with  the  Common  Criteria  is  that  although  it  allows 
product  evaluations  to  “mix  and  match”  assurance  processes,  in  practice  it  is  the  EAL 
collections  that  are  followed.  In  some  cases,  alternative  practices  might  actually  give 
more  assurance  than  the  processes  required  in  any  particular  EAL  package.  The  problem 
is  that  the  CC  drives  most  users  toward  accepting  only  a  particular  set  of  assurance  proc¬ 
esses  (those  in  the  EALs).  This  is  particularly  so  since  any  assurance  class  not  listed  in 
the  CC  (or  listed  beyond  EAL4)  will  not  be  mutually  recognized.  This  is  not  an  easy 
problem  to  fix;  it  is  often  difficult  to  measure  the  amount  of  actual  assurance  supplied  by 
different  assurance  processes  and,  as  a  result,  it  is  difficult  to  determine  when  replace¬ 
ment  of  one  by  another  is  acceptable.  However,  this  means  that  in  practice,  the  flexibility 
of  the  CC  in  terms  of  assurance  classes  is  often  underutilized. 

H.4  Options: 

1.  Make  no  changes. 

2.  Relax  requirements  in  the  CC  process  (especially  documentation). 

3.  Specifically  identify  alternatives  to  the  NIAP  process. 

4.  Specifically  create  and  identify  new  alternatives  to  the  NIAP  process,  espe¬ 
cially  for  low-assurance  evaluations. 

5.  Replace  the  NIAP  process  with  an  alternative  process. 

6.  Combine  new  approaches  for  low  assurance,  and  cost  reducers  for  high  assur¬ 
ance. 

H.5  Recommendation: 

We  recommend  a  cost  effective  alternative  for  low  assurance  evaluations  (EAL3  and  be¬ 
low).  Key  elements  of  this  low  assurance  evaluation  process  include;  process  assurance 
checking  for  the  developer,  minimal  functional  assurance  testing  by  a  laboratory,  and  the 
screening  of  the  code  and  execution  products  by  a  standard  set  of  tools. 
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Annex  J  NIAP  Estimates  to  Implement  Options 

The  following  tables  represent  the  independent  estimates  made  by  the  respective  partners  in  the  NIAP  - 
NSA  and  NIST  when  asked  to  gauge  the  costs  of  implementing  the  options  generated  after  synthesizing 
the  respective  policy,  practice,  and  expectations  assessments  of  the  review.  All  estimates  are  an  average  of 
estimates  independently  provided  by  the  NIST  and  NSA.  Dollars  are  in  millions. 

J.l  Option  1:  Eliminate  the  NIAP  and  product  evaluations 

No  estimate  was  provided  for  this  option.  However  it  should  be  noted  that  eliminating  the  current  costs  of 
NIAP  only  shifts  existing  internal  budgets  to  other  priorities  and  results  in  no  savings.  In  fact,  some  costs 
of  product  evaluations  may  shift  to  C&A  processes  or  result  in  these  costs  occurring  elsewhere  perhaps 
with  lesser  efficiencies. 


J.2  Option  2:  Continue  the  NIAP  in  its  current  form  (reduced  from  the  original  intent) 


FY07 

FY08 

FY09 

FY10 

FY11 

S8.233M 

S8.933M 

S9.433M 

$10.282M 

S11.270M 

Table  26.  Option  2  Cost  Estimate 

J.3  Option  3:  Restore  the  NIAP  to  the  original  intent  of  the  Letter  of  Partnership  between  NSA  and 
NIST 


FY07 

FY08 

FY09 

FY10 

FY11 

S16.190M 

S17.390M 

$19.515M 

$20.1 14M 

$25.530M 

Table  27.  Option  3  Cost  Estimate 

J.4  Option  4:  Modernize  the  approach  to  cybersecurity  evaluations  to  reflect  changes  in  the  envi¬ 
ronment  since  its  creation  in  1998 


FY07 

FY08 

FY09 

FY10 

FY11 

$19.865M 

$21.128M 

$24.8 14M 

$25.320M 

$27.358M 

Table  28.  Option  4  Cost  Estimate 

J.5  Option  5:  Integrated  approach  to  cybersecurity  evaluations 

NIST  and  NSA  were  unable  to  provide  cost  estimates  for  this  option. 

J.6  Option  6:  Forward  looking  approach  to  cybersecurity  evaluations  (new  paradigm) 

No  request  was  made  to  cost  this  option. 
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